Speech synthesis from neural decoding of spoken sentences

By Gopala Krishna Anumanchipalli, Josh Chartier, Edward F. Chang. Nature, 2019

869 citations recorded by the source corpus.

Technology that translates neural activity into speech would be transformative for people who are unable to communicate as a result of neurological impairments. Decoding speech from neural activity is challenging because speaking requires very precise and rapid multi-dimensional control of vocal tract articulators. Here we designed a neural decoder that explicitly leverages kinematic and sound representations encoded in human cortical activity to synthesize audible speech. Recurrent neural networks first decoded directly recorded cortical activity into representations of articulatory movement, and then transformed these representations into speech acoustics. In closed vocabulary tests, listeners could readily identify and transcribe speech synthesized from cortical activity. Intermediate articulatory dynamics enhanced performance even with limited data. Decoded articulatory representations were highly conserved across speakers, enabling a component of the decoder to be transferrable across participants. Furthermore, the decoder could synthesize speech when a participant silently mimed sentences. These findings advance the clinical viability of using speech neuroprosthetic technology to restore spoken communication. A neural decoder uses kinematic and sound representations encoded in human cortical activity to synthesize audible sentences, which are readily identified and transcribed by listeners.

Subfields: Cognitive Neuroscience

View Paper: https://www.semanticscholar.org/paper/3e271f66e696e68ada5ba1c41739c8d65426f71c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Semantic reconstruction of continuous language from non-invasive brain recordings (2022, methodology): “Unlike motor representations, which are only accessible during attempted speech 1,4 , semantic representations are accessible during both attempted and imagined speech.”
Machine Learning for Synthetic Data Generation: a Review (2023, background): “For example, in speech synthesis, the human evaluator rates the synthesized speech and real human speech in a blind manner [44], [205].”
High-performance brain-to-text communication via handwriting (2021, methodology): “Recently, speech-decoding BCIs have shown exciting promise for restoring rapid communication 16,17,34 , but their accuracies and vocabulary sizes require further improvement to support general-purpose use.”
An Integrated Brain-Machine Interface Platform With Thousands of Channels (2019, neutral): “For example, researchers have demonstrated human neuroprosthetic control of computer cursors [1, 2, 3], robotic limbs [4, 5], and speech synthesizers [6] using no more than 256 electrodes.”
Perspectives in conducting task-based research in pediatric surgical epilepsy patients (2026, unknown): “Such recordings in adults have been used to understand neural processing of speech [12–20], attention [21–25], vision [26], memory and spatial navigation [27–30], and for building brain computer interfaces [31–36].”
Distinct laminar origins of sensory-evoked high-gamma and low-frequency ECoG signals revealed by optogenetics (2026, unknown): “Enhancing our understanding of the sources of ECoG signals (76), especially from high - resolution, large - scale recording devices (77, 78), could enhance performance of neuroprosthetic systems (79) and diagnostic tools.”
Single-Neuron Recordings Research in Children: Ethical Considerations, Feasibility, Technical Aspects, and Scientific Opportunities (2025, unknown): “1), leading to, among many others, advances in our understanding of human language [2], memory [3, 4], sensorimotor function [5–7], psychiatric disease [8, 9], executive control [10–14], and brain-machine interfaces [15–18].”
Distinct laminar origins of high-gamma and low-frequency ECoG signals revealed by optogenetics (2025, unknown): “Enhancing our understanding of the sources of ECoG signals 73 , especially from high-resolution, large-scale recording devices 74,75 , could enhance performance of neuroprosthetic systems 76 and diagnostic tools.”
Dynamic Causal Tractography Analysis of Auditory Descriptive Naming: An Intracranial Study of 106 Patients (2025, unknown): “…EEG (iEEG) recording in patients with focal epilepsy can provide a unique opportunity to assess the dynamics of local cortical modulations during overt speech, as well as the neural coordination across cortical regions (Flinker et al., 2015; Anumanchipalli et al., 2019; Woolnough et al., 2021).”
A supervised data-driven spatial filter denoising method for speech artifacts in intracranial electrophysiological recordings (2023, unknown): “5,7).”