Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations

By Gökmen Tayfun, Yurii A. Vlasov. Frontiers in Neuroscience, 2016

441 citations recorded by the source corpus.

In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps∕s∕W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.

Subfields: Electrical and Electronic Engineering, Materials Chemistry

View Paper: https://www.semanticscholar.org/paper/c5d9c2d338f56441f6b9929dfa1c1ee97e0d8576

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

In Materials Chemistry

Citations in the corpus

Neuro-Inspired Computing With Emerging Nonvolatile Memorys (2018, background): “In principle, the weight update can be performed in a fully parallel fashion on the entire array as the programming voltages can be applied from both ends (row and column) to the synaptic device [80], [93].”
Ferroelectric FET analog synapse for acceleration of deep neural network training (2017, background): “Device requirements for accelerating the training of DNNs include ±1V, 1 nanosecond potentiation and depression programming pulses, a symmetric and linear conductance response with ≥32 conductance states (≥5-bit), and a Gmax/Gmin ratio of >10 [1][8].”
Hardware implementation of memristor-based artificial neural networks (2024, unknown): “The weight update process is implemented by applying voltage or current pulses to the memristors (block 13 and 14), following the Write-Verify (or Close Loop Tunning) 194 – 196 , or the Write-without-Verify (or Open Loop Tunning) 103,107,197,198 .”
A Review of Deep Learning Models for Time Series Prediction (2019, methodology): “For improving the computational efﬁciency, some studies have been conducted to propose algorithms and strategies for accelerating deep learning [221]–[223].”