EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras

By Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, Kostas Daniilidis. Robotics: Science and Systems Conference, 2018

537 citations recorded by the source corpus.

Event-based cameras have shown great promise in a variety of situations where frame based cameras suffer, such as high speed motions and high dynamic range scenes. However, developing algorithms for event measurements requires a new class of hand crafted algorithms. Deep learning has shown great success in providing model free solutions to many problems in the vision community, but existing networks have been developed with frame based images in mind, and there does not exist the wealth of labeled data for events as there does for images for supervised training. To these points, we present EV-FlowNet, a novel self-supervised deep learning pipeline for optical flow estimation for event based cameras. In particular, we introduce an image based representation of a given event stream, which is fed into a self-supervised neural network as the sole input. The corresponding grayscale images captured from the same camera at the same time as the events are then used as a supervisory signal to provide a loss function at training time, given the estimated flow from the network. We show that the resulting network is able to accurately predict optical flow from events only in a variety of different scenes, with performance competitive to image based networks. This method not only allows for accurate estimation of dense optical flow, but also provides a framework for the transfer of other self-supervised methods to the event-based domain.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/769ee0718f473e826f243812006011cf837a29db

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

ESIM: an Open Event Camera Simulator (2018, methodology): “Images from a standard stereo camera setup, ground truth camera poses, depth maps (derived from a LIDAR) and optic flow maps [15] are also provided.”
Learning to Detect Objects with a 1 Megapixel Event Camera (2020, background): “Even if these methods lose some of the event’s temporal resolution, they gain in terms of accuracy and scalability [24, 22].”
Event-Based Vision: A Survey (2019, methodology): “[22], [114] Full Dense ANN No Ye et al.”
High Speed and High Dynamic Range Video with an Event Camera (2019, unknown): “intermediate, “image-like” representations from event data [13], [14], [15], [15], [16].”
Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera (2018, background): “Although several works try to explore the advantages of the high temporal resolution provided by event cameras [39, 13, 26, 41, 40, 8, 15], how to make the best use of the event camera has not yet been fully investigated.”
Events-To-Video: Bringing Modern Computer Vision to Event Cameras (2019, methodology): “Following [55], we use skip connections between symmetric convolution layers.”