Backwards Differentiation in AD and Neural Nets: Past Links and New Opportunities

64 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Deep learning in neural networks: An overview (2014, methodology): “To my knowledge, the first NN-specific application of efficient BP as above was described in 1981 (Werbos, 1981, 2006).”
My First Deep Learning System of 1991 + Deep Learning Timeline 1962-2013 (2013, background): “Compare the concept of ordered derivative [89] and related work [28], with NN-speciﬁc discussion [89] (section 5.5.1), and the ﬁrst NN-speciﬁc efﬁcient BP of 1981 by Werbos [90, 92].”
Efficient robust policy optimization (2012, methodology): “Our gradient methods are analogous to reverse mode and reverse propagation of gradients in automatic/algorithmic differentiation, and the adjoint method [18], [19].”
Learning to Control PDEs with Differentiable Physics (2020, methodology): “, 2004) is used by most machine learning frameworks, where it is commonly known as reverse mode differentiation (Werbos, 2006; Chen et al., 2018).”
Value-gradient learning (2012, methodology): “Using the terminology of “automatic differentiation” (Rall, 1981; Werbos, 2005), this is called a reverse accumulation of the derivatives.”