Deep learning from crowds

By Filipe Rodrigues, Francisco Câmara Pereira. AAAI Conference on Artificial Intelligence, 2017

313 citations recorded by the source corpus.

Over the last few years, deep learning has revolutionized the field of machine learning by dramatically improving the state-of-the-art in various domains. However, as the size of supervised artificial neural networks grows, typically so does the need for larger labeled datasets. Recently, crowdsourcing has established itself as an efficient and cost-effective solution for labeling large sets of data in a scalable manner, but it often requires aggregating labels from multiple noisy contributors with different levels of expertise. In this paper, we address the problem of learning deep neural networks from crowds. We begin by describing an EM algorithm for jointly learning the parameters of the network and the reliabilities of the annotators. Then, a novel general-purpose crowd layer is proposed, which allows us to train deep neural networks end-to-end, directly from the noisy labels of multiple annotators, using only backpropagation. We empirically show that the proposed approach is able to internally capture the reliability and biases of different annotators and achieve new state-of-the-art results for various crowdsourced datasets across different settings, namely classification, regression and sequence labeling.

Subfields: Artificial Intelligence, Computer Science Applications, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/a464e45d17da3e60bffb87290fab46f89607b7be

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Toward Annotator Group Bias in Crowdsourcing (2021, unknown): “Following the traditional probabilistic approaches for label aggregation (Raykar et al., 2010; Rodrigues and Pereira, 2018; Li et al., 2019), we propose a novel framework GroupAnno that models the production of annotations as a stochastic process via a novel probabilistic graphical model (PGM).”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, background): “stention [90], estimate noise rate [91], [92], similarity loss [93], transfer learning [94], -distribution [95] 6. Labeler Quality Assessment EM [25], [96], [97], trace regularizer [98], crowd-layer [99], image difﬁculty estimate [100], consistency with network prediction [101], omitting probability”
A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis (2019, background): “Rodrigues and Pereira (2018) describe an approach to assess the reliability of annotators in a crowd, and a crowd layer used to train deep models from noisy labels from multiple annotators, internally capturing the reliability and biases of di ﬀ erent annotators to achieve state-of-the-art results…”
Co-teaching: Robust training of deep neural networks with extremely noisy labels (2018, background): “[35] added a crowd layer after the output layer for noisy labels from multiple annotators.”
How does Disagreement Help Generalization against Label Corruption? (2019, background): “[31] added a crowd layer after the output layer for noisy labels from multiple annotators.”