"Who" Is Saying "What"? Brain-Based Decoding of Human Voice and Speech

By Elia Formisano, Federico De Martino, M. Bonte, Rainer Goebel. Science, 2008

589 citations recorded by the source corpus.

Can we decipher speech content (“what” is being said) and speaker identity (“who” is saying it) from observations of brain activity of a listener? Here, we combine functional magnetic resonance imaging with a data-mining algorithm and retrieve what and whom a person is listening to from the neural fingerprints that speech and voice signals elicit in the listener's auditory cortex. These cortical fingerprints are spatially distributed and insensitive to acoustic variations of the input so as to permit the brain-based recognition of learned speech from unknown speakers and of learned voices from previously unheard utterances. Our findings unravel the detailed cortical layout and computational properties of the neural populations at the basis of human speech recognition and speaker identification.

Subfields: Cognitive Neuroscience, Signal Processing

View Paper: https://www.semanticscholar.org/paper/0af2fa5ea40018fbf3a02498f0e3648459af275a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Signal Processing

Citations in the corpus

How can we measure awareness? An overview of current methods (2015, background): “However, neither does such an instrument exists nor can we conceive of any way of building it (though some are trying; Haynes and Rees 2005; Kamitani and Tong 2005; Formisano et al. 2008; Haynes 2009).”
Reconstructing Speech from Human Auditory Cortex (2012, background): “Previous work suggests that categorical representations are an important organizational principle in STG [9,45–47].”
Sparse logistic regression for whole brain classification of fMRI data (2010, background): “…as they identified a number of key auditory structures thought to be sensitive to both acoustical differences in the posterior temporal cortex (Formisano et al., 2008; Tervaniemi et al., 2006), as well as areas within the anterior temporal (Humphries et al., 2005; Rogalsky and Hickok, 2008)…”
Processing of Natural Sounds in Human Auditory Cortex: Tonotopy, Spectral Tuning, and Relation to Voice Sensitivity (2012, background): “” This choice was based on the observations that (1) in natural settings processing of speech necessarily involved processing of voice, and (2) overlapping regions in the superior temporal cortex are involved in both speech and voice processing (Formisano et al., 2008; Petkov et al., 2009).”
The Encoding of Speech Sounds in the Superior Temporal Gyrus (2019, background): “Indeed, there is evidence of non-linear encoding of acoustic input across neural populations that encode acoustic-phonetic features, which corresponds to categorical phoneme percepts (Chang et al., 2010; Evans and Davis, 2015; Formisano et al., 2008; Lee et al., 2012).”
Decoding neural representational spaces using multivariate pattern analysis. (2014, background): “…work has shown that MVP classiﬁcation can also distinguish many other brain states, for example low-level visual features in the early visual cortex (Haynes & Rees 2005, Kamitani & Tong 2005) and auditory stimuli in the auditory cortex (Formisano et al. 2008, 436 Haxby · Connolly · Guntupalli”
A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy. (2018, unknown): “The extent and nature of hierarchical organization in humans has thus remained unsettled (Formisano et al., 2008; Leaver and Rauschecker, 2010; Staeren et al., 2009; Wessinger et al., 2001).”