Unbiased Online Recurrent Optimization

By Corentin Tallec, Yann Ollivier. 2017

107 citations recorded by the source corpus.

The novel Unbiased Online Recurrent Optimization (UORO) algorithm allows for online learning of general recurrent computational graphs such as recurrent network models. It works in a streaming fashion and avoids backtracking through past activations and inputs. UORO is computationally as costly as Truncated Backpropagation Through Time (truncated BPTT), a widespread algorithm for online learning of recurrent networks. UORO is a modification of NoBackTrack that bypasses the need for model sparsity and makes implementation easy in current deep learning frameworks, even for complex models. Like NoBackTrack, UORO provides unbiased gradient estimates; unbiasedness is the core hypothesis in stochastic gradient descent theory, without which convergence to a local optimum is not guaranteed. On the contrary, truncated BPTT does not provide this property, leading to possible divergence. On synthetic tasks where truncated BPTT is shown to diverge, UORO converges. For instance, when a parameter has a positive short-term but negative long-term influence, truncated BPTT diverges unless the truncation span is very significantly longer than the intrinsic temporal range of the interactions, while UORO performs well thanks to the unbiasedness of its gradients.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Decoupled Neural Interfaces using Synthetic Gradients (2016, methodology): “While Real-Time Recurrent Learning (Williams & Zipser, 1989) or approximations such as (Ollivier & Charpiat, 2015; Tallec & Ollivier, 2017) may seem a promising way to remove update locking, these methods require maintaining the full (or approximate) gradient of the current state with respect to…”
Sparse Attentive Backtracking: Long-Range Credit Assignment in Recurrent Networks (2017, methodology): “The Unbiased Online Recurrent Optimization (UORO) (Tallec & Ollivier, 2017) method gets around this by updating a rank-1 approximation to this gradient tensor, which is shown to keep the estimate of the gradient unbiased, but potentially at the risk of increasing the variance of the gradient estimat”
Unbiasing Truncated Backpropagation Through Time (2017, background): “Backpropagation Through Time (BPTT) Werbos (1990) is the de facto standard for training recurrent neural networks.”
Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation (2016, methodology): “An encouraging direction is that proposed by Ollivier et al. (2015); Tallec and Ollivier (2017), which shows that computationally efﬁcient estimators of the gradient can be obtained using a forward method (online estimation of the gradient), which avoids the need to store all past states in…”
A solution to the learning dilemma for recurrent networks of spiking neurons (2019, methodology): “But interesting approximations to RTRL have subsequently been pro424 posed (see [47] for a review): some stochastic approximations [48] which are O(n(3)) or only 425 applicable for small networks [49], and also recently two deterministic O(n(2)) approxima426 tions [50, 51].”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, methodology): “However, the push for continuously learning systems that can run indefinitely long has cast a spotlight back on RTRL (and variants [194], [195], [196], [197], [198]), with a focus on improving computational and memory efficiency.”