Machine translation of cortical activity to text with an encoder–decoder framework

By Joseph G. Makin, David A. Moses, Edward F. Chang. Nature Neuroscience, 2019

315 citations recorded by the source corpus.

A decade after speech was first decoded from human brain signals, accuracy and speed remain far below that of natural speech. Here we show how to decode the electrocorticogram with high accuracy and at natural-speech rates. Taking a cue from recent advances in machine translation, we train a recurrent neural network to encode each sentence-length sequence of neural activity into an abstract representation, and then to decode this representation, word by word, into an English sentence. For each participant, data consist of several spoken repeats of a set of 30–50 sentences, along with the contemporaneous signals from ~250 electrodes distributed over peri-Sylvian cortices. Average word error rates across a held-out repeat set are as low as 3%. Finally, we show how decoding with limited data can be improved with transfer learning, by training certain layers of the network under multiple participants’ data. Makin and colleagues decode speech from neural signals recorded during a preoperative procedure, using an algorithm inspired by machine translation. For one participant reading from a closed set of 50 sentences, decoding accuracy is nearly perfect.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/bf20b9f5dba74f4ca43e5bb450c937a5caf792ca

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

  1. Krishna V. Shenoy
  2. Leigh R. Hochberg
  3. Jaimie M. Henderson
  4. Francis R. Willett
  5. Donald T. Avansino
  6. John P. Cunningham
  7. John P. Donoghue

In Electrical and Electronic Engineering

  1. Krishna V. Shenoy

Citations in the corpus