Learning Policies with External Memory

By Leonid Peshkin, Nicolas Meuleau, Leslie Pack Kaelbling. 1999

In order for an agent to perform well in partially observable domains, it is usually necessary for actions to depend on the history of observations. In this paper, we explore a {\it stigmergic} approach, in which the agent's actions include the ability to set and clear bits in an external memory, and the external memory is included as part of the input to the agent. In this case, we need to learn a reactive policy in a highly non-Markovian domain. We explore two algorithms: SARSA(\lambda), which has had empirical success in partially observable domains, and VAPS, a new algorithm due to Baird and Moore, with convergence guarantees in partially observable domains. We compare the performance of these two algorithms on benchmark problems.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Memory-based control with recurrent neural networks (2015, background): “[32] instead take the view of [17] and consider memory as extra state dimensions that can can be read and set by the policy.”
Learning from Scarce Experience (2002, methodology): “Recently, stochastic gradient descent methods, in particular REINFORCE (Williams, 1992; Williams & Peng, 1991), have been used in conjunction with policy classes constrained in various ways: with external memory (Peshkin et al., 1999), finite state controllers (Meuleau et al.”
Learning to Cooperate via Policy Search (2000, background): “There are many possibilities for constructing policies with memory [13, 10].”
Learning deep neural network policies with continuous memory states (2015, background): “In fact, prior work has proposed using discrete memory with storage actions [3].”
Reinforcement Learning with Factored States and Actions (2004, background): “There has been comparatively little on dealing with largeaction spaces or factored actions (Dean et al., 1998; Meuleau et al., 1998; Peshkin et al.,1999; Guestrin et al., 2002).”
Reinforcement Learning with Long Short-Term Memory (2001, background): “The second alternative is a table-based system extended with memory bits that are part of the observation, and that the controller can switch on and o [9].”
Learning Finite-State Controllers for Partially Observable Environments (1999, result): “We conclude with empirical results comparing the performance of stochastic and exact gradient descent, and showing the ability of our algorithm to extract the useful information contained in the sequence of past observations to compensate for the lack of observability at each time-step.”
Learning latent structure: Carving nature at its joints (2010, background): “Recently, Todd et al. [44] showed how the gating framework can be interpreted as a policy search algorithm [45].”
Policy Learning with Continuous Memory States for Partially Observed Robotic Control (2015, background): “While most policy search methods are concerned with reactive policies, a number of methods have been proposed that augment the policy with internal state, including methods based on finite state controllers [9, 14] and explicit memory states that the policy can alter using memory storage actions [11”
Policy-Gradient Algorithms for Partially Observable Markov Decision Processes (2003, methodology): “This includes IState-GPOMDP and Istate Williams’ REINFORCE [Peshkin et al., 1999].”