Equivalent-accuracy accelerated neural-network training using analogue memory

By S. Ambrogio, Pritish Narayanan, H. Tsai, R. Shelby, I. Boybat, C. D. Nolfo, Severin Sidler, M. Giordano, Martina Bodini, Nathan C. P. Farinha and 4 more. Nature, 2018

974 citations recorded by the source corpus.

Neural-network training can be slow and energy intensive, owing to the need to transfer the weight data for the network between conventional digital memory chips and processor chips. Analogue non-volatile memory can accelerate the neural-network training algorithm known as backpropagation by performing parallelized multiply–accumulate operations in the analogue domain at the location of the weight data. However, the classification accuracies of such in situ training using non-volatile-memory hardware have generally been less than those of software-based training, owing to insufficient dynamic range and excessive weight-update asymmetry. Here we demonstrate mixed hardware–software neural-network implementations that involve up to 204,900 synapses and that combine long-term storage in phase-change memory, near-linear updates of volatile capacitors and weight-data transfer with ‘polarity inversion’ to cancel out inherent device-to-device variations. We achieve generalization accuracies (on previously unseen data) equivalent to those of software-based training on various commonly used machine-learning test datasets (MNIST, MNIST-backrand, CIFAR-10 and CIFAR-100). The computational energy efficiency of 28,065 billion operations per second per watt and throughput per area of 3.6 trillion operations per second per square millimetre that we calculate for our implementation exceed those of today’s graphical processing units by two orders of magnitude. This work provides a path towards hardware accelerators that are both fast and energy efficient, particularly on fully connected neural-network layers. Analogue-memory-based neural-network training using non-volatile-memory hardware augmented by circuit simulations achieves the same accuracy as software-based training but with much improved energy efficiency and speed.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/3a2345fcb4d2e2fe4572123a438bc8f30a8d93ae

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Fully hardware-implemented memristor convolutional neural network (2020, methodology): “The most recent studies report the demonstrations of two-layer 4 and three-layer 27 memristor multi-layer perceptronsfor image recognition using the MNIST (Modified National Institute of Standards and Technology) handwritten-digit database 10 .”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “For instance, using RRAM [336], PCRAM [337], and memristors, the MVM operation can be performed fast in the analog domain within one clock cycle.”
An artificial spiking afferent nerve based on Mott memristors for neurorobotics (2020, background): “This poses a signi cant challenge to the traditional computing architecture due to the von-Neumann bottleneck 3 – 5 .”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Taking memristor (e.g., RRAM [268], PCRAM [252]) as an example, the MVM can be performed in the analog domain.”
A compute-in-memory chip based on resistive random-access memory (2022, positive): “Although these studies proposed various techniques to mitigate the impacts of analogue-related hardware non-idealities on inference accuracy, the AI benchmark results reported were often obtained by performing software emulation based on characterized device data 3,5,21,24 .”
Hardware implementation of memristor-based artificial neural networks (2024, positive): “This is the most commonly used approach on research prototypes developed as technology demonstrators due to its versatility, as the activation function can be implemented and changed by simply modifying the software code 55,100,113 – 115, 122, 123 .”