A Theory of Adaptive Pattern Classifiers

By Shun-ichi Amari. 1967

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “…space of complex, nonlinear, differentiable (Leibniz, 1684), multi-stage, NN-related systems has been discussed at least since the early 1960s (e.g., Amari, 1967; Bryson, 1961; Bryson & Denham, 1961; Bryson & Ho, 1969; Director & Rohrer, 1969; Dreyfus, 1962; Kelley, 1960; Pontryagin, Boltyanskii,…”
DeltaGrad: Rapid retraining of machine learning models (2020, neutral): “See e.g., (Gladyshev, 1965; Amari, 1967; Kul’chitskiy & Mozgovoy, 1992; Bert-sekas & Tsitsiklis, 1996; Moulines & Bach, 2011; Karimi et al., 2016; Bottou et al., 2018; Gorbunov et al., 2019; Gower et al., 2019), etc, and references therein.”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “Fortunately, various authors (Murata, 1998; Bottou and LeCun, 2005; Bordes et al., 2009) building on early work of Amari (1967), provide some further insight into this question by studying asymptotic behavior of general iterations of the form18 θk+1 = θk − αkB−1k gk(θk) , (19) where Bk = B is a…”
Parallelized Stochastic Gradient Descent (2010, background): “While there is considerable research regarding the asymptotic properties of this process [1, 8], not much is known regarding the number of iterations required until the asymptotic regime is assumed.”
On-line learning and stochastic approximations (1999, background): “Almost all of the early work on Learning Systems focused on online algorithms (Hebb, 1949) (Rosenblatt, 1957) (Widrow and Hoff, 1960) (Amari, 1967) (Kohonen, 1982).”
Adaptive Online Learning Algorithms for Blind Separation: Maximum Entropy and Minimum Mutual Information (1997, methodology): “The performances of the algorithms (A′) with equation 5.1 will be compared with that of the algorithms (A) without adaptation in the next section by simulation.”
Action and Perception as Divergence Minimization (2020, background): “This is typical in supervised learning where the test set is unavailable during training (Bishop, 2006), in online learning where training inputs become available over time (Amari, 1967), and in ﬁltering where only inputs up to the current time are available (Kalman, 1960).”
Discriminative learning in sequential pattern recognition (2008, methodology): “The MCE-based classifier design is a discriminant-function-based approach to pattern recognition [1], [28], [29].”
Neural networks and neuroscience-inspired computer vision. (2014, methodology): “In the 1960s and 1970s, the back-propagation algorithm [26–28] was introduced, which provided a concrete mechanism for propagating error signals back through a multi-layer neural network.”
Natural Gradient Works Efficiently in Learning (1998, methodology): “where ·t is a learning rate that may depend on t and C.w/ is a suitably chosen positive definite matrix (see Amari, 1967 ).”
Dynamics of Learning Near Singularities in Layered Networks (2008, methodology): “This average is justified by the stochastic approximation when source signals (y t , x t ) are ergodic (Amari, 1967, 1977).”
Gradient-based learning applied to document recognition (1998, background): “The properties of such algorithms applied to learning have been studied theoretically since the 1960's [9], [10], [11], but practical successes for non-trivial tasks did not occur until the mid eighties.”
Merging chrominance and luminance in early, medium, and late fusion using Convolutional Neural Networks (2015, methodology): “Amari31 used the stochastic gradient method to train a pattern classification model in his publication “A theory of adaptive pattern classifiers” [Amari, 1967].”
Neural Networks and the Bias/Variance Dilemma (1992, background): “For a precise technical definition of Vapnikzervonenkis dimension (and some generalizations), as well as for demonstrations of its utility in establishing uniform convergence results, the reader is referred to Vapnik (1982), Pollard (1984), Dudley (1987), and Haussler (1989a).”
Change-Point Detection in Time-Series Data by Direct Density-Ratio Estimation (2009, background): “Our idea basically relies on stochastic gradient descent [1], which allows us to learn the parameters efficiently in an online manner.”
Asymptotic statistical theory of overtraining and cross-validation (1997, methodology): “This training procedure is usually carried out by a gradient descent method minimizing a target function ([ 1 ], [27], and many others).”
Hidden Markov processes (2002, unknown): “See also Amari [8].”