Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation

By Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J. Guibas, Andrea Tagliasacchi, Frank Dellaert, Thomas A. Funkhouser. Computer Vision and Pattern Recognition, 2022

330 citations recorded by the source corpus.

We present Panoptic Neural Fields (PNF), an object-aware neural scene representation that decomposes a scene into a set of objects (things) and background (stuff). Each object is represented by an oriented 3D bounding box and a multi-layer perceptron (MLP) that takes position, direction, and time and outputs density and radiance. The background stuff is represented by a similar MLP that additionally outputs semantic labels. Each object MLPs are instance-specific and thus can be smaller and faster than previous object-aware approaches, while still leveraging category-specific priors incorporated via meta-learned initialization. Our model builds a panoptic radiance field representation of any scene from just color images. We use off-the-shelf algorithms to predict camera poses, object tracks, and 2D image semantic segmentations. Then we jointly optimize the MLP weights and bounding box parameters using analysis-by-synthesis with self-supervision from color images and pseudo-supervision from predicted semantic segmentations. During experiments with real-world dynamic scenes, we find that our model can be used effectively for several tasks like novel view synthesis, 2D panoptic segmentation, 3D scene editing, and multiview depth prediction.

Subfields: Aerospace Engineering, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d4a265b6008058506a143177422fe192e4fe2090

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “For specific domains like traffic scenes [25, 41], we may instead train a closed-set segmentation model and use its prediction for training object-aware neural fields.”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, neutral): “We compare our method with the latest SOTA approaches trained on the KITTI360, including NeRF-based PNF [6] and 3D Gaussian-based 3DGS [4].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, methodology): “These methods can produce more realistic sensor data visually than graphics-based approaches, but they have limitations such as high rendering times or requiring independent training for each scene being reconstructed [107], [115], [116], [117], [118].”
ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes (2023, methodology): “3D Instance Segmentation We compare 3 methods for 3D instance segmentation on ScanNet++: PointGroup [16], HAIS [7] and SoftGroup [44] on 75 instances classes - semantic classes excluding wall, ceiling and floor.”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, methodology): “Building upon it, subsequent research focusing on in-domain scene modeling, has expanded the scope by introducing instance modeling [9, 20, 39, 47] or encoding visual features [17, 35, 44–46] that support semantic delineation.”