Brain Function and Adaptive Systems: A Heterostatic Theory

By A. Harry Klopf. 1972

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

RUDDER: Return Decomposition for Delayed Rewards (2018, background): “g), A a0(:) = P a0ˇ(a 0js t+1)(:) (expected SARSA), A a0(:) sample a0from ˇ(a0js t+1) (SARSA). The next theorem states that TD has an exponential decay for Q-value updates even for eligibility traces [56,6,106,99]. Theorem 2. For initialization q0(s t;a t) = 0 and delayed reward with r t = 0 for t 6”
Reinforcement Learning with Replacing Eligibility Traces (2005, methodology): “The second mechanism that has been widely used for handling delay is the el-igibility trace.1 Introduced by Klopf (1972), eligibility traces have been used in avariety of reinforcement learning systems (e.g., Barto, Sutton & Anderson, 1983;Lin, 1992; Tesauro, 1992; Peng & Williams, 1994).”
Reinforcement learning with replacing eligibility traces (2004, background): “1 Introduced by Klopf (1972), eligibility traces have been used in a variety of re-inforcement learning systems (e.g., Barto, Sutton & Anderson, 1983; Lin, 1992; Tesauro, 1992; Peng & Williams, 1994).”
Recent Advances in Hierarchical Reinforcement Learning (2003, background): “Other early RL research was explicitly motivated by animal behavior and its neural basis [45, 33, 34, 71].”
Reinforcement learning (1998, background): “Klopf hypothesized that neurons, the major components of our brains, are individually hedonists that work to maximize a local analog of pleasure while minimizing a local analog of pain (Klopf 1972, 1982).”
Hierarchical Error Representation: A Computational Model of Anterior Cingulate and Dorsolateral Prefrontal Cortex (2015, methodology): “Rather than training weights using the stimulus vector s, however, an eligibility trace vector d is used instead (Barto, Sutton, & Anderson, 1983; Klopf, 1972).”
Landmark learning: An illustration of associative search (1981, unknown): “An ASN incorporates learning rules that have been carefully designed following Klopfs hypothesis that neurons are goalseeking systems (Klopf, 1972, 1979, 1980).”
Learning to Act Using Real-Time Dynamic Programming (1995, methodology): “Submitted to AI Journal special issue on Computational Theories of Interaction and Agency the proposals of Klopf [36; 37], Sutton and Barto [70; 71; 72] developed these methods as models of animal learning.”
The convergence of TD(λ) for general λ (1992, methodology): “Many other proposals along similar lines have been made: Sutton acknowledged the influence of Klopf (1972; 1982) and in Sutton (1988) discussed Holland's bucket brigade method for classifier systems (Holland, 1986), and a procedure by Witten (1977).”
Time-Derivative Models of Pavlovian Reinforcement (1990, methodology): “Klopf (1988) actually proposed a time course of eligibility that was inverted-U shaped,”
Cognitive Control (2012, unknown): “Although TD learning has its early roots in animal psychology and artificial intelligence [71], [72], the first algorithm, called TD(0), and the famous example of random walk was created by Sutton [73] (likewise the term TD).”
Adaptive Critics and the Basal Ganglia (1994, unknown): “Following Klopf (1972, 1982), we say that input activity at t ¡ 1 (i.e., x it ¡ 1 6 = 0) makes the connection weight v i eligible for modiﬂcation at t .”
A comparison of natural and artificial intelligence (1975, methodology): “Such a theory has been NATURAL vs ARTIFICIAL described in detail in Klopf (1972) and in summary form in Klopf (1974).”
Intrinsic Motivation and Reinforcement Learning (2013, background): “Because RL addresses how predictive values biological RL despite what it borrows from, and suggests about, biological RL.”
Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications (2018, background): “Almost two decades later, Klopf [51] integrated the mechanism of temporal-difference (TD) learning from psychology into the computational model of TE learning.”
Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules (2018, methodology): “…1992; Schultz, 1998) but in some of the early studies it remained unclear whether eligibility traces can be set by presynaptic activity alone (Klopf, 1972; Sutton and Barto, 1981) or only by Hebbian co-activation of pre- and postsynaptic neurons (Barto et al., 1983; Barto, 1985; Williams,…”