Environment-Independent Task Specifications via GLTL

By Michael L. Littman, Ufuk Topcu, Jie Fu, Charles Lee Isbell Jr., Min Wen, James MacGlashan. 2017

162 citations recorded by the source corpus.

We propose a new task-specification language for Markov decision processes that is designed to be an improvement over reward functions by being environment independent. The language is a variant of Linear Temporal Logic (LTL) that is extended to probabilistic specifications in a way that permits approximations to be learned in finite time. We provide several small environments that demonstrate the advantages of our geometric LTL (GLTL) language and illustrate how it can be used to specify standard reinforcement-learning tasks straightforwardly.

Subfields: Computer Networks and Communications, Hardware and Architecture, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Programmable Agents (2017, neutral): “The declarative paradigm provides an appealing and flexible way to describe tasks for agents [23].”
Learning to Compose Skills (2017, unknown): “), eventually ( ♦ ), and until ( U ), along with the basic logic connectives: negation( ¬ ), disjunction ( ∨ ), conjunction ( ∧ ) and implication ( → ) [14].”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “However, it is worth noting that certain preferences, such as some temporal logic statements, cannot be expressed as a reward function without augmenting the state space; some cannot be expressed by any reward function at all (Littman et al., 2017).”
Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning (2020, methodology): “1992b) or linear temporal logic (Li et al., 2017; Littman et al., 2017; Toro Icarte et al., 2018b; Hasanbeig et al., 2018; Camacho et al., 2019; De Giacomo et al., 2019; Shah et al., 2020) – and then generate a reward function towards fulfilling that specification.”
Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning (2018, background): “Recently, there has been significant interest in using Linear Temporal Logic (LTL) to specify tasks in RL (Li et al., 2017; Littman et al., 2017; Toro Icarte et al., 2018; Hasanbeig et al., 2018).”
Deep Reinforcement Learning from Policy-Dependent Human Feedback (2019, unknown): “Due to the non-Markovian nature of the task (Bacchus et al., 1996; Littman et al., 2017), it is uninformative to report summaries of Markovian rewards; accordingly, we report two complementary statistics that, together, illustrate the success (or failure) of an agent in achieving the desired motion.”
LTL2Action: Generalizing LTL Instructions for Multi-Task RL (2021, methodology): “This paper builds on past work in RL which explores using LTL (or similar formal languages) for reward function specification, decomposition, or shaping (e.g., Aksaray et al., 2016; Li et al., 2017; Littman et al., 2017; Toro Icarte et al., 2018b; Li et al., 2018; Camacho et al., 2017; 2019; Yuan et”