High Speed and High Dynamic Range Video with an Event Camera

By Henri Rebecq, René Ranftl, Vladlen Koltun, Davide Scaramuzza. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019

815 citations recorded by the source corpus.

Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous “events” instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our quantitative experiments show that our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (<inline-formula><tex-math notation="LaTeX">$>\!20\%$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>></mml:mo><mml:mspace width="-0.166667em"/><mml:mn>20</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="rebecq-ieq1-2963386.gif"/></alternatives></inline-formula>), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (<inline-formula><tex-math notation="LaTeX">$>5,000$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>></mml:mo><mml:mn>5</mml:mn><mml:mo>,</mml:mo><mml:mn>000</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="rebecq-ieq2-2963386.gif"/></alternatives></inline-formula> frames per second) of high-speed phenomena (e.g., a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. As an additional contribution, we demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data. We release the reconstruction code, a pre-trained model and the datasets to enable further research.

Subfields: Electrical and Electronic Engineering, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/630c9e9594ae79f82bea1c55ed7eb6db52f0e47f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

  1. Davide Scaramuzza
  2. Daniel Gehrig
  3. Guillermo Gallego
  4. Kostas Daniilidis
  5. Tobi Delbruck
  6. Henri Rebecq
  7. Garrick Orchard
  8. Andrew J. Davison

In Radiology, Nuclear Medicine and Imaging

  1. Davide Scaramuzza
  2. Guillermo Gallego
  3. Kostas Daniilidis
  4. Amos Sironi
  5. Tobi Delbruck
  6. Henri Rebecq

Citations in the corpus