Reviving and Improving Recurrent Back-Propagation

By Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, Kijung Yoon, Xaq Pitkow, Raquel Urtasun, Richard S. Zemel. 2018

140 citations recorded by the source corpus.

In this paper, we revisit the recurrent back-propagation (RBP) algorithm, discuss the conditions under which it applies as well as how to satisfy them in deep neural networks. We show that RBP can be unstable and propose two variants based on conjugate gradient on the normal equations (CG-RBP) and Neumann series (Neumann-RBP). We further investigate the relationship between Neumann-RBP and back propagation through time (BPTT) and its truncated version (TBPTT). Our Neumann-RBP has the same time complexity as TBPTT but only requires constant memory, whereas TBPTT's memory cost scales linearly with the number of truncation steps. We examine all RBP variants along with BPTT and TBPTT in three different application domains: associative memory with continuous Hopfield networks, document classification in citation networks using graph neural networks and hyperparameter optimization for fully connected networks. All experiments demonstrate that RBPs, especially the Neumann-RBP variant, are efficient and effective for optimizing convergent recurrent neural networks. Code is released at: \url{this https URL}.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Graph Normalizing Flows (2019, methodology): “A recent approach for memory saving based on recurrent backpropagation (RBP) [1, 20, 18] requires running message passing to convergence, followed by the approximate, iterative inversion of a large matrix.”
Incremental Few-Shot Learning with Attention Attractor Networks (2018, methodology): “We also show how this regularizer can be optimized, using recurrent back-propagation (Liao et al., 2018; Almeida, 1987; Pineda, 1987) we show empirically that our proposed method can produce state-of-the-art results in incremental few-shot learning on the challenging mini -ImageNet (Vinyals et al.,…”
Residual Flows for Invertible Generative Modeling (2019, neutral): “On the other hand, our approach shares some similarities with Recurrent Backpropagation (Almeida, 1987; Pineda, 1987; Liao et al., 2018), since both approaches leverage convergent dynamics to modify the derivatives.”
Deep Equilibrium Models (2019, neutral): “Recent work [28] has re-examined RBP and established an implicit, constant-memory variant based on conjugate gradient and Neumann series.”
JFB: Jacobian-Free Backpropagation for Implicit Networks (2021, methodology): “To apply the Jacobian-based backpropagation in (13), we use the conjugate gradient (CG) method on an associated set of normal equations similarly to (Liao et al. 2018).”
SIN: Semantic Inference Network for Few-Shot Streaming Label Learning (2022, background): “Attention attractor network (ANN) [17] trains a set of new weights to recognize new labels until they converge, using recurrent back-propagation [33].”
Multiscale Deep Equilibrium Models (2020, background): “Pineda [43] and Almeida [2] studied implicit differentiation techniques for training recurrent dynamics, also known as recurrent back-propagation (RBP) [37].”
Convolutional Bipartite Attractor Networks (2019, background): “Recent explorations have considered stronger training procedures [e.g., 26, 42]; however, as for all recurrent nets, training is complicated by the issue of vanishing/exploding gradients.”
Bilevel Optimization: Convergence Analysis and Enhanced Design (2020, methodology): “, 2005; Moore, 2010) and gradientbased methods (Domke, 2012; Pedregosa, 2016; Gould et al., 2016; Maclaurin et al., 2015; Franceschi et al., 2018; Ghadimi & Wang, 2018; Liao et al., 2018; Shaban et al., 2019; Hong et al., 2020; Liu et al., 2020; Li et al., 2020; Grazzi et al., 2020; Lorraine et al.,”