Adaptive Switching Circuits

By Tifatfj Lassified, .. I, Ad., Iivices Technical, Informahon Agenq, A. Hall, Eiectroiiiis Mborrtories, ITMItRI MIViiilTV. 2016

984 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/85d346297ec37a451d4500afad53fed8f281af92

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Online Extreme Evolutionary Learning Machines (2014, unknown): “At each iteration t, the network is presented with a single observation (x (t) , y (t) ) from a noisy data stream and the output weights w are updated in order to reduce the mean squared error between the observed target value y (t) and the predicted valuê y (t) by means of the Delta-rule (Widrow an”
Excess-Risk of Distributed Stochastic Learners (2013, background): “…T k,iw) (1) while a mean-square-error algorithm would minimize the expected value of the quadratic loss [28] (also referred to as the “delta rule” [29]): QLMS(w,hk,i,yk(i)) , ρ 2 ‖w‖2 + (yk(i)− hTk,iw)2 (2) The expectation of the loss function over the distribution of the data is referred…”
Switching between internal and external modes: A multiscale learning principle (2017, background): “These algorithms become more powerful, however, when the learning process is further driven by a mismatch, or error, signal (Rosenblatt, 1958; Widrow & Hoff, 1960).”
Linear Transformers Are Secretly Fast Weight Programmers (2021, methodology): “Therefore, inspired by recent work on FWPs (Schlag et al., 2021), we introduce an improved programming instruction akin to the famous error-correcting delta-rule (Widrow & Hoff, 1960).”
Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training (2015, unknown): “(10) Specifically, this update rule is called the adaline algorithm [43], used in adaptive signal processing and control [44].”
Neural Networks and Deep Learning (2018, unknown): “The Widrow-Hoff algorithm was proposed in [531] and is closely related to Tikhonov-Arsenin’s work [499].”
Ieee Transactions on Knowledge and Data Engineering 1 Multi-label Neural Networks with Applications to Functional Genomics and Text Categorization (unknown): “Earliest work on neural networks dates back to McCulloch and Pitts's M-P model of a neuron [19], which is then followed by considerable works in the 1950s and 1960s on single-layer neural networks [23] [31].”
Learning functions across many orders of magnitudes (2016, background): “An important special case is when fθ is a neural network (McCulloch and Pitts, 1943; Widrow and Hoff, 1960; Rosenblatt, 1962; Rumelhart et al., 1986; Bishop, 1995).”
Transformers learn in-context by gradient descent (2022, methodology): “To provide evidence for this hypothesis, we build on Schlag et al. (2021) that already provide a linear self-attention layer variant with (fast-)inner loop learning by the error-correcting delta rule (Widrow & Hoff, 1960).”
Adaptive Networks (2014, background): “During the 1950s, stochastic approximation theory did not receive much attention in the engineering community until the landmark work by [39], which developed the real form of the LMS algorithm [see (13f)].”
Bidirectional Helmholtz Machines (2015, background): “The origin of the term “neural network” can be traced back to the work of trying to model the neuron in biological systems [28, 66, 111].”
Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, unknown): “DeltaNet uses the delta update rule [ 114] to operationalize this mechanism.”
Survey on SVM and their application in image classification (2018, unknown): “Square (LMS) algorithm [30].”
Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, unknown): “The delta update rule (Widrow et al., 1960; Schlag et al., 2021b) dynamically erases the value ( v old t ) associated with the current input key ( k t ) and writes a new value ( v new t ), which is a linear combination of the current input value and the old value based on the “writing strength” β t…”