End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging

By Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen P. Boyd, Wolfgang Heidrich, Felix Heide, Gordon Wetzstein. ACM Transactions on Graphics, 2018

432 citations recorded by the source corpus.

In typical cameras the optical system is designed first; once it is fixed, the parameters in the image processing algorithm are tuned to get good image reproduction. In contrast to this sequential design approach, we consider joint optimization of an optical system (for example, the physical shape of the lens) together with the parameters of the reconstruction algorithm. We build a fully-differentiable simulation model that maps the true source image to the reconstructed one. The model includes diffractive light propagation, depth and wavelength-dependent effects, noise and nonlinearities, and the image post-processing. We jointly optimize the optical parameters and the image processing algorithm parameters so as to minimize the deviation between the true and reconstructed image, over a large set of images. We implement our joint optimization method using autodifferentiation to efficiently compute parameter gradients in a stochastic optimization algorithm. We demonstrate the efficacy of this approach by applying it to achromatic extended depth of field and snapshot super-resolution imaging.

Subfields: Biomedical Engineering, Biophysics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/72e46dcb57eceb74df19f91bb5bb529cbe8e98ae

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Biophysics

In Computer Vision and Pattern Recognition

Citations in the corpus

Inference in artificial intelligence with deep optics and photonics (2020, methodology): “…in red whenever the amplitude terms are most relevant and blue whenever the phase terms dominate. freeform lenses for extended depth of field 65 , image classification 13–15 , flat cameras 72 , high dynamic range imaging 73 , wavelength demultiplexing 74 , or depth sensing with conventional…”
Deep Learning Techniques for Inverse Problems in Imaging (2020, background): “A series of recent works have modeled cameras as differentiable optical systems and then used back-propagation to design specialized optical filters and diffractive optical elements for improved demosaicing [173], color imaging and demosaicing [173], super-resolution and extended depth-of-field imag”