The Thing that we Tried Didn't Work very Well: Deictic Representation in Reinforcement Learning
By Sarah Finney, Natalia Hernandez-Gardiol, Leslie Pack Kaelbling, Tim Oates. 2002
42 citations recorded by the source corpus.
Most reinforcement learning methods operate on propositional representations of the world state. Such representations are often intractably large and generalize poorly. Using a deictic representation is believed to be a viable alternative: they promise generalization while allowing the use of existing reinforcement-learning methods. Yet, there are few experiments on learning with deictic representations reported in the literature. In this paper we explore the effectiveness of two forms of deictic representation and a naive propositional representation in a simple blocks-world domain. We find, empirically, that the deictic representations actually worsen learning performance. We conclude with a discussion of possible causes of these results and strategies for more effective learning in domains with objects.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Lesia Semenova
- Chudi Zhong
- Zhi Chen
- Haiyang Huang
- Cynthia Rudin
- Balaraman Ravindran
- Andrew Barto
- Robert W. Platt
Citations in the corpus
- Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “Typically, in relational reinforcement learning, policies are represented as relational regression trees [90, 91, 143, 96, 76] that are more interpretable than neural networks.”
- Solving Relational and First-Order Logical Markov Decision Processes: A Survey (2012, background): “Some works use deictic representations (e.g. Finney et al, 2002) in which propositional encodings are used to refer to objects.”
- Learning multi-agent state space representations (2010, unknown): “An agent-centric representation only has meaning relative to the agent using it and to the context in which it is being used [5].”
- Deictic Image Maps: An Abstraction For Learning Pose Invariant Manipulation Policies (2018, background): “The resulting system was recognized to be partially observable and a variety of methods were explored for combating the problem without much success [15, 16, 17].”
- An algebraic approach to abstraction in reinforcement learning (2004, methodology): “The various approaches to learning with deictic pointers (Whitehead and Ballard, 1991; Finney et al., 2002) usually employ simple pointers similar to those in Figure 6.”