Unsupervised Video Object Segmentation for Deep Reinforcement Learning

By Vikrant Goel, James Weng, Pascal Poupart. 2018

72 citations recorded by the source corpus.

We present a new technique for deep reinforcement learning that automatically detects moving objects and uses the relevant information for action selection. The detection of moving objects is done in an unsupervised way by exploiting structure from motion. Instead of directly learning a policy from raw images, the agent first learns to detect and segment moving objects by exploiting flow information in video sequences. The learned representation is then used to focus the policy of the agent on the moving objects. Over time, the agent identifies which objects are critical for decision making and gradually builds a policy based on relevant moving objects. This approach, which we call Motion-Oriented REinforcement Learning (MOREL), is demonstrated on a suite of Atari games where the ability to detect moving objects reduces the amount of interaction needed with the environment to obtain a good policy. Furthermore, the resulting policy is more interpretable than policies that directly map images to actions or values with a black box neural network. We can gain insight into the policy by inspecting the segmentation and motion of each object detected by the agent. This allows practitioners to confirm whether a policy is making decisions based on sensible information.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised Learning of Object Keypoints for Perception and Control (2019, methodology): “Both instance and semantic segmentation has been approached using supervised [22, 25] and unsupervised learning [2, 9, 14, 10, 16, 21, 6] methods.”
Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning (2019, background): “Greydanus et al. (2017) generate saliency maps by perturbing the original input image using a Gaussian blur of the image and measure changes in policy from removing information from a region.”
Image Segmentation Using Deep Learning: A Survey (2020, methodology): “For example, MOREL [188] introduced a deep reinforcement learning approach formoving object segmentation in videos.”
MASK-RL: Multiagent Video Object Segmentation Framework Through Reinforcement Learning (2020, background): “For these reasons, RL has recently attracted a lot of attention for emulating human behavior in several contexts, such as games [21], [29] and autonomous driving [22], but also for computer vision tasks [1], [13], [19], [25], [27], [47], including video object segmentation [11], [12], [37].”
Exploiting Spatial Invariance for Scalable Unsupervised Object Tracking (2019, methodology): “Examples include (Greff, van Steenkiste, and Schmidhuber 2017; van Steenkiste et al. 2018; Goel, Weng, and Poupart 2018; Greff et al. 2019; Burgess et al. 2019).”
Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards (2019, methodology): “This failure can be resolved by extracting the entities from the states using semantic segmentation [Goel et al., 2018], then applying our method upon the semantic segmented entities instead of each grid.”
Adaptive Auxiliary Task Weighting for Reinforcement Learning (2019, methodology): “Optical Flow [14] Given two consecutive visual observations, predict the optical flow.”
Entity Abstraction in Visual Model-Based Reinforcement Learning (2019, background): “But processing a disentangled latent state with a single function [3, 19–22] or processing each disentangled factor in a permutation-sensitive manner [3, 11, 23] (1) assumes a ﬁxed number of entities that cannot be dynamically adjusted for generalizing to more objects than in training and (2) has…”