Deep Learning Human Mind for Automated Visual Classification

By Concetto Spampinato, Simone Palazzo, Isaak Kavasidis, Daniela Giordano, Nasim Souly, Mubarak Shah. 2016

314 citations recorded by the source corpus.

What if we could effectively read the mind and transfer human visual capabilities to computer vision methods? In this paper, we aim at addressing this question by developing the first visual object classifier driven by human brain signals. In particular, we employ EEG data evoked by visual object stimuli combined with Recurrent Neural Networks (RNN) to learn a discriminative brain activity manifold of visual categories in a reading the mind effort. Afterward, we transfer the learned capabilities to machines by training a Convolutional Neural Network (CNN)–based regressor to project images onto the learned manifold, thus allowing machines to employ human brain–based features for automated visual classification. We use a 128-channel EEG with active electrodes to record brain activity of several subjects while looking at images of 40 ImageNet object classes. The proposed RNN-based approach for discriminating object classes using brain signals reaches an average accuracy of about 83%, which greatly outperforms existing methods attempting to learn EEG visual object representations. As for automated object categorization, our human brain–driven approach obtains competitive performance, comparable to those achieved by powerful CNN models and it is also able to generalize over different visual datasets.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Training Networks in Null Space of Feature Covariance for Continual Learning (2021, background): “Deep neural networks have achieved promising performance on various tasks in natural language processing, machine intelligence, etc., [5, 9, 48, 49, 52].”
Deep learning-based electroencephalography analysis: a systematic review (2019, background): “The result which achieved an accuracy difference of nearly 77% [179] was found to be caused by a flawed design in [107] and should therefore be considered as an outlier.”
Deep learning for healthcare applications based on physiological signals: A review (2018, background): “, 2017 [101] Discriminate brain activity CNN Six subject were shown 2000 images in 4 sessions Accuracy 86.”
Cogni-Net: Cognitive Feature Learning Through Deep Visual Perception (2018, methodology): “Unlike [8], our work attempts to utilize the link between the visual stimuli (images) and cognitive activity (EEG signals), taking inspiration from [9] which attempts to learn acoustic representations from unlabeled videos.”
EEG-Based Brain-Computer Interfaces (BCIs): A Survey of Recent Studies on Signal Sensing Technologies and Computational Intelligence Approaches and Their Applications (2020, methodology): “A framework aimed at addressing visual object classification was proposed by [136], which applied an RNN to EEG data invoked by visual stimulation.”
Multi-view Adversarially Learned Inference for Cross-domain Joint Distribution Matching (2018, methodology): “The ImageNet-EEG dataset [25] involves six subjects who were shown images of objects while Electroencephalogram (EEG) data was recorded.”