Reinforcement Learning with Soft State Aggregation

By Satinder Singh, Tommi S. Jaakkola, Michael I. Jordan. 1994

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “Furthermore, state aggregation techniques, whether based on bisimulation or other methods (Abel et al., 2017; Li et al., 2006; Singh et al., 1995; Givan et al., 2003; Jiang et al., 2015; Ruan et al., 2015), suffer from poor compatibility with function approximation methods.”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, methodology): “Classically, interpolating these two extremes were semi-batch RL algorithms (Lange et al., 2012; Singh et al., 1995), which improve the policy through repetitions of collecting a large batch of transitions D = { ( s, a, s (cid:48) , r ) } and performing many or full policy updates.”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, methodology): “Utilizing representation learning within model-based RL has been studied in a number of previous works [23], including using embeddings for state aggregation [36], dimensionality reduction [31], self-organizing maps [37], value prediction [32], and deep auto-encoders [22, 11, 43, 15].”
An analysis of reinforcement learning with function approximation (2008, methodology): “A is the induced (easily extendable to our setting) the proof of Theorem 1 operator norm and the one in the de.nition of .b is in (Tsitsiklis & Van Roy, 1996a).”
Experience Replay for Real-Time Reinforcement Learning Control (2012, methodology): “Hence, they are widely used in approximate RL, under various names such as “interpolative representations” [34], “soft aggregation” [27], and “fuzzy approximation” [15].”
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning (2018, unknown): “, 2018), including using embeddings for state aggregation (Singh et al., 1994), dimensionality reduction (Nouri & Littman, 2010), self-organizing maps (Smith, 2002), value prediction (Oh et al.”
Efficient Exploration and Value Function Generalization in Deterministic Systems (2013, result): “Our results here add to the literature on agnostic reinforcement learning with such a hypothesis class [21, 25, 7, 26].”
Reinforcement Learning: An Introduction (1998, methodology): “The theory of state aggregation in reinforcement learning has been developed by Singh, Jaakkola, and Jordan (1995) and Tsitsiklis and Van Roy (1996).”
A Unified Analysis of Value-Function-Based Reinforcement-Learning Algorithms (1999, unknown): “3.1”
Policy Error Bounds for Model-Based Reinforcement Learning with Factored Linear Models (2016, background): “In hard aggregation, R is a point evaluator, while in soft aggregation (Singh et al., 1995) it is stochastic.”
An analysis of temporal-difference learning with function approximation (1997, methodology): “The m o r e g e n eral case, involving t he u s e of function approximation, is addressed by r e s u l ts i n (Dayan, 1992), (Tsitsiklis and V an Roy, 1996), (Gordon, 1995), and (Singh et al., 1995).”
Finite time bounds for sampling based fitted value iteration (2005, unknown): “[Research Report] 2007, pp.46.”
Finite-Time Bounds for Fitted Value Iteration (2008, result): “…and Van Roy (1996) considered the planning scenario with known dynamics and making use of a set of representative states, subsequent results by Singh et al. (1995), Ormoneit and Sen (2002) and Szepesvári and Smart (2004) considered less restricted problem settings, though none of these…”
Batch Reinforcement Learning (2012, background): “Another class is formed by the 'semi-batch' algorithms that were introduced in the 90's (Singh et al, 1995) primarily for formal reasons.”