From Skills to Symbols: Learning Symbolic Representations for Abstract High-Level Planning

By George Dimitri Konidaris, Leslie Pack Kaelbling, Tomas Lozano-Perez. 2018

361 citations recorded by the source corpus.

We consider the problem of constructing abstract representations for planning in high-dimensional, continuous environments. We assume an agent equipped with a collection of high-level actions, and construct representations provably capable of evaluating plans composed of sequences of those actions. We first consider the deterministic planning case, and show that the relevant computation involves set operations performed over sets of states. We define the specific collection of sets that is necessary and sufficient for planning, and use them to construct a grounded abstract symbolic representation that is provably suitable for deterministic planning. The resulting representation can be expressed in PDDL, a canonical high-level planning domain language; we construct such a representation for the Playroom domain and solve it in milliseconds using an off-the-shelf planner. We then consider probabilistic planning, which we show requires generalizing from sets of states to distributions over states. We identify the specific distributions required for planning, and use them to construct a grounded abstract symbolic representation that correctly estimates the expected reward and probability of success of any plan. In addition, we show that learning the relevant probability distributions corresponds to specific instances of probabilistic density estimation and probabilistic classification. We construct an agent that autonomously learns the correct abstract representation of a computer game domain, and rapidly solves it. Finally, we apply these techniques to create a physical robot system that autonomously learns its own symbolic representation of a mobile manipulation task directly from sensorimotor data---point clouds, map locations, and joint angles---and then plans using that representation. Together, these results establish a principled link between high-level actions and abstract representations, a concrete theoretical foundation for constructing abstract representations with provable properties, and a practical mechanism for autonomously learning abstract high-level representations.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “Using symbolic representations for state and action is also an inherent characteristic of works in task-and-motionplanning (Kaelbling and Lozano-Pérez, 2011; Konidaris et al., 2018) and symbolic planning (Asai and Fukunaga, 2017).”
Learning First-Order Symbolic Representations for Planning from the Structure of the State Space (2019, methodology): “ methods for learning abstract planning representations [7]. Other recent methods produce PDDL models from given macro-actions (options) but these models are propositional and hence do not generalize [24]. Deep reinforcement learning (DRL) methods [30], on the other hand, generate policies over high”
Relational inductive biases, deep learning, and graph networks (2018, background): “, 2018), developing model-based approaches with an emphasis on abstraction (Kansky et al., 2017; Konidaris et al., 2018; Zhang et al., 2018; Hay et al., 2018), investing more heavily in meta-learning (Wang et al.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, background): “Along this direction, researchers have studied building symbolic representations for skills (Konidaris et al., 2018) and learning instruction semantics from interaction (Oh et al., 2017)”
Learning Plannable Representations with Causal InfoGAN (2018, methodology): “[21] automatically extracted state representations from raw observations, but relied on a prespecified set of skills for the task.”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, background): “[82] present an action-centric approach to symbolic planning.”
Value Preserving State-Action Abstractions (2020, background): “…levels of granularity (Dayan and Hinton, 1993; Wiering and Schmidhuber, 1997; Parr and Russell, 1998; Dietterich, 2000a; Barto and Mahadevan, 2003; Jong and Value Preserving State-Action Abstractions Stone, 2008; Konidaris, 2016; Bai and Russell, 2017; Konidaris et al., 2018; Levy et al., 2019).”
An empirical investigation of the challenges of real-world reinforcement learning (2020, background): “In addition, research has been done to develop a symbolic planning language that could be useful for explainability [Konidaris et al. 2018; James et al. 2018].”
Learning Markov State Abstractions for Deep Reinforcement Learning (2021, background): “A full investigation of other representation-learning properties (e.g. value preservation (Abel et al., 2016), symbol construction (Konidaris et al., 2018), suitability for planning (Kurutach et al., 2018), information compression (Abel et al., 2019)) is beyond the scope of this paper.”
Composable Planning with Attributes (2018, background): “In (Konidaris et al., 2018), the authors propose a model that learns the symbols for a STRIPS-style representation.”
What Should I Do Now? Marrying Reinforcement Learning and Symbolic Planning (2019, background): “[21] attempt to learn symbolic representations by interacting with an environment and observing the effects of actions.”
HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators (2019, background): “Konidaris et al. [39] learns symbolic representations by interacting with an environment and observing the effects of the actions.”
Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems (2021, neutral): “Object-oriented symbolic interfaces like these have also been used in other previous works to allow humans to provide informative and generalizable object-focused ad-vices in an effortless way (Thomaz, Breazeal et al. 2006; Krening et al. 2016; Guan et al. 2021).”
Inventing Relational State and Action Abstractions for Effective and Efficient Bilevel Planning (2022, background): “While previous works enforce the downward reﬁnability property [8, 12, 13, 14], it is important in robotics to be robust to violations of this property, since learned abstractions will typically lose critical geometric information.”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “We hope to Approaches such as symbolic planning [3], hierarchical RL [4, 5, 6, 7, 8], and hierarchical planning [9] can accomplish compositional task generalization by reasoning in an abstract or symbolic space.”
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning (2023, neutral): “In classical planning, the community has explored numerous learning-based methods [59, 61, 9, 25, 4] and interactive editor-based methods [49] for acquiring symbolic domain models.”
Text2Motion: from natural language instructions to feasible plans (2023, methodology): “[18] grounds an LLM planner to admissible action sets for task planning, [19, 20] explore the integration of LLMs with PDDL [1], and [21, 22] focuses on task-level replanning with LLMs. Tangential works shift the representation of plans from action sequences to code [5, 23–25] and embed task queries”
Context-Specific Representation Abstraction for Deep Option Learning (2021, background): “Konidaris, Kaelbling, and Lozano-Perez (2018) introduces the importance of abstraction in planning problems with Chitnis et al. (2020) performing a contextspecific abstraction for the purposes of decomposition in planning-related tasks.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, methodology): “The grounding or anchoring of a predicate refers to the mapping between the (often continuous) low-level state and context and the predicate’s truth value; it defines the meaning of the symbol [208, 70].”