Sleep spindle detection: crowdsourcing and evaluating performance of experts, non-experts, and automated methods

By S. Warby, S. Wendt, Peter Welinder, E. Munk, O. Carrillo, H. Sørensen, Poul J. Jennum, Paul E. Peppard, Pietro Perona, E. Mignot. Nature Methods, 2014

365 citations recorded by the source corpus.

Sleep spindles are discrete, intermittent patterns of brain activity observed in human electroencephalographic data. Increasingly, these oscillations are of biological and clinical interest because of their role in development, learning and neurological disorders. We used an Internet interface to crowdsource spindle identification by human experts and non-experts, and we compared their performance with that of automated detection algorithms in data from middle- to older-aged subjects from the general population. We also refined methods for forming group consensus and evaluating the performance of event detectors in physiological data such as electroencephalographic recordings from polysomnography. Compared to the expert group consensus gold standard, the highest performance was by individual experts and the non-expert group consensus, followed by automated spindle detectors. This analysis showed that crowdsourcing the scoring of sleep data is an efficient method to collect large data sets, even for difficult tasks such as spindle identification. Further refinements to spindle detection algorithms are needed for middle- to older-aged subjects.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Physiology

View Paper: https://www.semanticscholar.org/paper/36cbbf484415d27760286f5b937a9954fe82622d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

Citations in the corpus

U-Time: A Fully Convolutional Network for Time Series Segmentation Applied to Sleep Staging (2019, background): “…work shows that such systems can be highly robust (even compared to human performance) and may play an important role in developing novel biomarkers for sleep disorders and other (e.g., neurodegenerative and psychiatric) diseases [Stephansen et al., 2018, Warby et al., 2014, Schenck et al., 2014].”
U-Sleep: resilient high-frequency sleep staging (2021, neutral): “While manual sleep staging follows guidelines as suggested by, for example, the AASM 16 , it is a dif ﬁ cult process with room for interpretation, making it inconsistent and error prone 19 .”
Extracting spatial-temporal coherent patterns in large-scale neural recordings using dynamic mode decomposition. (2014, background): “Recently, a number of these algorithms were evaluated against experts and crowd-sourced spindle detectors (Warby et al., 2014).”
Rotating waves during human sleep spindles organize global patterns of activity that repeat precisely through the night (2016, methodology): “The number of spindles detected was in agreement with previous reports of spindle density (Gais et al., 2002; Warby et al., 2014).”
Targeting neural oscillations with transcranial alternating current stimulation (2021, background): “Sleep spindles are a well-known example of bursts of rhythmic activity that are critical for the consolidation of new memories (Schabus, 2004; Warby, 2014).”