Robust high-dimensional memory-augmented neural networks

By Geethan Karunaratne, Manuel Schmuck, M. Le Gallo, Giovanni Cherubini, Luca Benini, Abu Sebastian, Abbas Rahimi. 2020

123 citations recorded by the source corpus.

Traditional neural networks require enormous amounts of data to build their complex mappings during a slow training procedure that hinders their abilities for relearning and adapting to new data. Memory-augmented neural networks enhance neural networks with an explicit memory to overcome these issues. Access to this explicit memory, however, occurs via soft read and write operations involving every individual memory entry, resulting in a bottleneck when implemented using the conventional von Neumann computer architecture. To overcome this bottleneck, we propose a robust architecture that employs a computational memory unit as the explicit memory performing analog in-memory computation on high-dimensional (HD) vectors, while closely matching 32-bit software-equivalent accuracy. This is achieved by a content-based attention mechanism that represents unrelated items in the computational memory with uncorrelated HD vectors, whose real-valued components can be readily approximated by binary, or bipolar components. Experimental results demonstrate the efficacy of our approach on few-shot image classification tasks on the Omniglot dataset using more than 256,000 phase-change memory devices. Our approach effectively merges the richness of deep neural network representations with HD computing that paves the way for robust vector-symbolic manipulations applicable in reasoning, fusion, and compression. The implementation of memory-augmented neural networks using conventional computer architectures is challenging due to a large number of read and write operations. Here, Karunaratne, Schmuck et al. propose an architecture that enables analog in-memory computing on high-dimensional vectors at accuracy matching 32-bit software equivalent.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Materials Chemistry

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Materials Chemistry

Citations in the corpus

Memristive technologies for data storage, computation, encryption, and radio-frequency communication (2022, background): “However, although a conventional CAM finds an exact match between the input and stored data, it cannot compute the degree of match for each data entry with high precision (70).”
A TinyML Platform for On-Device Continual Learning With Quantized Latent Replays (2021, methodology): “Finally, several online-learning devices using bioinspired algorithms such as Spiking Neural Networks [52] and High-Dimensional Computing [25] have been proposed [53]– [55].”
Generalized Key-Value Memory to Flexibly Adjust Redundancy in Memory-Augmented Networks (2022, methodology): “A recent methodology [12] addresses these issues by enhancing the key memory representations with robust properties of hyperdimensional computing [13], such that the representations can be readily transformed to low-precision (i.e., bipolar or binary) vectors in the key memory while exhibiting…”