A Simple Method to Determine if a Music Information Retrieval System is a “Horse”

By Bob L. Sturm. 2014

160 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Speech & Audio

In Signal Processing

Citations in the corpus

Learning Visual N-Grams from Web Data (2016, background): “As a result, models such as ours may not necessarily achieve state-of-the-art results on established benchmarks, because they did not learn to exploit the biases of those benchmarks as well [25, 51, 52].”
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (2016, background): “Statistical learning systems, like those used for VQA, may develop similar “cheating” approaches to superficially “solve” tasks without learning the underlying reasoning processes [35, 36].”
Compositional Attention Networks for Machine Reasoning (2018, background): “(2016) show how the first generation of successful models on VQA tasks tend to acquire only superficial comprehension of both the image and the question, exploiting dataset biases rather than capturing a sound perception and reasoning process that would lead to the correct answer (Sturm, 2014).”
Invariant Risk Minimization (2019, background): “While machines are able to learn complex prediction rules by minimizing their training error, data are often marred by selection biases, confounding factors, and other peculiarities [49, 48, 23].”
Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients (2017, background): “It is unlikely that regularizing for this kind of smoothness will be a panacea for all manifestations of the “Clever Hans” effect (Sturm 2014) in deep neural networks, but in this case the prior it represents – that predictions should not be sensitive to small perturbations in input space – helps…”
Local Interpretable Model-Agnostic Explanations for Music Content Analysis (2017, methodology): “Variants of this approach include saliency maps [20], explanation vectors [1], “horse” detection [22] and local interpretable model-agnostic explanations (LIME) [17].”
A dataset and architecture for visual reasoning with a working memory (2018, background): “There has been strong suggestion in the literature that networks trained on these datasets focus on statistical regularities for the prediction tasks, whereby a system may “cheat” to supericially solve a given task [11, 10].”
Psychology Meets Machine Learning: Interdisciplinary Perspectives on Algorithmic Job Candidate Screening (2018, background): “…the more descriptive aspects of such phenomena (e.g. Collobert and Interdisciplinary Perspectives on Algorithmic Job Candidate Screening 211 Weston 2008; Hamel and Eck 2010), it is still problematic for them to capture notions of true human-like ‘understanding’ (Hofstadter 2018; Sturm 2014).”