Learning Continuous Image Representation with Local Implicit Image Function

By Yinbo Chen, Sifei Liu, Xiaolong Wang. Computer Vision and Pattern Recognition, 2020

970 citations recorded by the source corpus.

How to represent an image? While the visual world is presented in a continuous manner, machines store and see the images in a discrete way with 2D arrays of pixels. In this paper, we seek to learn a continuous representation for images. Inspired by the recent progress in 3D reconstruction with implicit neural representation, we propose Local Implicit Image Function (LIIF), which takes an image coordinate and the 2D deep features around the coordinate as inputs, predicts the RGB value at a given coordinate as an output. Since the coordinates are continuous, LIIF can be presented in arbitrary resolution. To generate the continuous representation for images, we train an encoder with LIIF representation via a self-supervised task with superresolution. The learned continuous representation can be presented in arbitrary resolution even extrapolate to ×30 higher resolution, where the training tasks are not provided. We further show that LIIF representation builds a bridge between discrete and continuous representation in 2D, it naturally supports the learning tasks with size-varied image ground-truths and significantly outperforms the method with resizing the ground-truths. Our project page with code is at https://yinboc.github.io/liif/.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/767a6054796e2e6c1de453afab0e05e55aadf825

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

3D Neural Field Generation Using Triplane Diffusion (2022, neutral): “Moreover, GANs can be difficult to train and suffer from issues like mode collapse [75] whereas diffusion models train stably and have been shown to better capture the full training distribution.”
WIRE: Wavelet Implicit Neural Representations (2023, background): “Consequently, INRs have advanced the state of the art in numerous vision-related areas, including computer graphics [22, 27, 28], image processing [10], inverse problems [42], and signal representations [41].”
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis (2021, background): “With MLPs, however, pixel-wise learnable upsamplers such as pixelshufﬂe (Shi et al., 2016) or LIIF (Chen et al., 2021b) produce “chessboard” or “texture sticking” artifacts due to its tendency of relying on the image coordinates implicitly.”
PETR: Position Embedding Transformation for Multi-View 3D Object Detection (2022, methodology): “In MetaSR [17] and LIFF [8], the HR coordinates are encoded into the LR features and HR images of arbitrary size can be generated.”
Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning (2022, methodology): “These representations are used to upsample the images either to specific scales [37] or to arbitrary resolutions [10, 30, 58].”
Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction (2023, background): “Compared with explicit representations like voxel and BEV, implicit representations usually share the advantage of arbitrary-resolution modeling and computation-efﬁcient architectures [6, 11, 40].”
InfinityGAN: Towards Infinite-Resolution Image Synthesis (2021, methodology): “Note that using spatially-dependent variables as inputs in a neural implicit function is less common, but similar formulations can be found in [3, 31].”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, background): “Local implicit representations [3, 5, 23, 56] and hybrid explicit–implicit representations [11,35,39,53] combine the beneﬁts of both types of representations by offering computationally and memory-efﬁcient architectures.”
Alias-Free Generative Adversarial Networks (2021, methodology): “Recent implicit network [51, 54, 13] based GANs [4, 52] generate each pixel independently via similar 1×1 convolutions.”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, background): “smooth latent-space interpolations and unconditional shape samples [4, 21, 27].”