Reinforcement Learning in Markovian and Non-markovian Environments

By urgen Schmidhuber. 1991

Subfields: Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

Recurrent World Models Facilitate Policy Evolution (2018, methodology): “Many basic concepts ﬁrst explored in the 1980s for feed-forward neural networks (FNNs) [56, 58, 72, 104, 105] and in the 1990s for RNNs [74, 75, 76, 78] laid some of the groundwork for Learning to Think [83].”
Continuous Deep Q-Learning with Model-based Acceleration (2016, methodology): “As in standard RL, model-based deep reinforcement learning methods have generally been more efﬁcient (Li & Todorov, 2004; Watter et al., 2015; Li & Todorov, 2004; Wahlstr ¨ om et al., 2015; Levine & Koltun, 2013), while model-free algorithms tend to be more generally applicable but substantially slo”
Deep learning in neural networks: An overview (2014, methodology): “To allow for memories of previous events in partially observable worlds (Section 6.3), the most general variant of this technique uses RNNs instead of FNNs to implement bothM and C (Feldkamp & Puskorius, 1998; Schmidhuber, 1990d, 1991c).”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “This is known as “backpropagation through a model” [22, 27, 28, 42, 33, 4, 5].”
World Models (2018, methodology): “Using RNNs to develop internal models to reason about the future has been explored as early as 1990 in a paper called Making the World Differentiable (Schmidhuber, 1990a), and then further explored in (Schmidhuber, 1990b; 1991a; 1990c).”
Reinforcement Learning: A Survey (1996, positive): “…several reinforcement learning algorithms including the interval exploration method (Kaelbling, 1993b) (described shortly), the exploration bonus in Dyna (Sutton, 1990), curiosity-driven exploration (Schmidhuber, 1991a), and the exploration mechanism in prioritized sweeping (Moore & Atkeson, 1993).”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “This triggered work on partially observable Markov decision problems (POMDPs) [223, 222, 227, 204, 205, 206, 316, 148, 278, 122, 152, 25, 114, 160, 126, 308, 309, 183].”
One Big Net For Everything (2018, methodology): “However, reinforcement learning RNN controllers can proﬁt from gradient-based RNNs used as predictive world models [55, 56, 60, 77].”
Algorithms for Sequential Decision Making (1996, background): “This approach has been studied by a number of researchers [106, 89, 137].”
Optimal Artiﬁcial Curiosity, Creativity, Music, and the Fine Arts (2005, background): “…[44,34], there is just one life, no need for predeﬁned repeatable trials, no restriction to Markovian interfaces between sensors and environment [25], and the utility function implicitly takes into account the expected remaining lifespan E µ ( T | h ( ≤ t )) and thus the possibility to extend…”
Goedel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements (2003, background): “Or the interface to the environment is Markovian [35], that is, the current input always uniquely identiﬁes the environmental state—a lot of work has already been done on this special case [33, 2, 60].”
Gödel Machines: Fully Self-referential Optimal Universal Self-improvers (2007, background): “Or the interface to the environment is Markovian [33], that is, the current input always uniquely identifies the environmental state—a lot of work has been done on this special case [31] [2] [55].”
Probabilistic Incremental Program Evolution (1997, background): “The EFs are exploited in an on-line fashion to learn rewarding action sequences (Whitehead & Ballard, 1990; Schmidhuber, 1991; Chrisman, 1992; Lin, 1993; Clii & Ross, 1994; Ring, 1995; McCallum, 1996; Wiering & Schmidhuber, 1996a).”
Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010) (2010, methodology): “a) A traditional Q-learner [117] able to deal with delayed rewards as long as the environment is fully observable, like in Section III-B? A more limited 1-step look-ahead learner [52] that will break down in presence of delayed intrinsic rewards? A more sophisticated RL algorithm for delayed rewards”
Hidden-Mode Markov Decision Processes for Nonstationary Sequential Decision Making (2001, background): “Proceedings of the IEEE, 77(2), Feb.J. H. Schmidhuber (1990).”
Reinforcement Learning Upside Down: Don't Predict Rewards - Just Map Them to Actions (2019, methodology): “The earlier systems [42, 43, 45] also use gradient-based SL in RNNs to learn mappings from costs/rewards and other inputs to actions.”
Curious model-building control systems (1991, methodology): “Extensions of this approach [9] rely on the same basic principles.”