Closed-Loop Learning of Visual Control Policies

By Sébastien Jodogne, Justus H. Piater. 2011

43 citations recorded by the source corpus.

In this paper we present a general, flexible framework for learning mappings from images to actions by interacting with the environment. The basic idea is to introduce a feature-based image classifier in front of a reinforcement learning algorithm. The classifier partitions the visual space according to the presence or absence of few highly informative local descriptors that are incrementally selected in a sequence of attempts to remove perceptual aliasing. We also address the problem of fighting overfitting in such a greedy algorithm. Finally, we show how high-level visual features can be generated when the power of local descriptors is insufficient for completely disambiguating the aliased states. This is done by building a hierarchy of composite features that consist of recursive spatial combinations of visual features. We demonstrate the efficacy of our algorithms by solving three visual navigation tasks and a visual version of the classical "Car on the Hill" control problem.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Robotics & Control

Citations in the corpus

• End-to-End Training of Deep Visuomotor Policies (2015, neutral): “ include full torque control of 7-DoF robotic arms interacting with objects, with 30-40 state dimensions. In simple synthetic environments, control from images has been addressed with image features (Jodogne and Piater, 2007), nonparametric methods (van Hoof et al., 2015), and unsupervised state-spa”
• On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, neutral): “2) [42] used as sensory preprocessors to create less redundant sensory codes [118, 138, 142, 46].”
• Deep predictive policy training using reinforcement learning (2017, neutral): “The study conducted by Jodogne and Piater [15] is among”
• Learning visual representations for perception-action systems (2011, unknown): “We therefore followed the first approach and introduced a method called Reinforcement Learning of Visual Classes (RLVC) that adaptively and incrementally discretizes a continuous or very large discrete perceptual space into discrete states (Jodogne and Piater 2005a, 2007).”
• Learning Visual Representations for Interactive Systems (2009, unknown): “We therefore followed the first approach and introduced a method called Reinforcement Learning of Visual Classes (RLVC) that adaptively and incrementally discretizes a continuous or very large discrete perceptual space into discrete states [8, 11].”
• Evolving deep unsupervised convolutional networks for vision-based reinforcement learning (2014, unknown): “…of single-agent RL (i.e. TD [23], policy gradients [22], etc.), is to combine action learning with an unsupervised learning (UL) preprocessor or “compressor” which provides a lower-dimensional feature vector that the agent receives as input instead of the raw observation [5, 8, 11, 15, 17, 18, 19].”
• Deep auto-encoder neural networks in reinforcement learning (2010, unknown): “The interesting work of Jodogne et al. [10] can be seen at the verge of fully integrating the learning of the image processing into RL.”
• Construction of approximation spaces for reinforcement learning (2013, unknown): “Current approaches to visual tasks aim thus to learn a feature mapping φ : Z → IR p from observations z, without loosing too much information about the true state x (see Jodogne and Piater, 2007, for an overview).”
• Online Evolution of Deep Convolutional Network for Vision-Based Reinforcement Learning (2014, unknown): “), is to combine action learning with an unsupervised learning (UL) preprocessor or “compressor” which provides a lower-dimensional feature vector that the agent receives as input instead of the raw observation [5, 8, 12, 17, 19–21].”
• State-of-the-Art in Visual Attention Modeling (2013, unknown): “Based on the idea that vision serves action, Jodogne [162] introduced an approach for learning action-based image classification known as Reinforcement Learning of Visual Classes (RLVC).”