FIERY: Future Instance Prediction in Bird’s-Eye View from Surround Monocular Cameras

By Anthony Hu, Zak Murez, Nikhil C. Mohan, Sof'ia Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, Alex Kendall. IEEE International Conference on Computer Vision, 2021

344 citations recorded by the source corpus.

Driving requires interacting with road agents and predicting their future behaviour in order to navigate safely. We present FIERY: a probabilistic future prediction model in bird’s-eye view from monocular cameras. Our model predicts future instance segmentation and motion of dynamic agents that can be transformed into non-parametric future trajectories. Our approach combines the perception, sensor fusion and prediction components of a traditional autonomous driving stack by estimating bird’s-eye-view prediction directly from surround RGB monocular camera inputs. FIERY learns to model the inherent stochastic nature of the future solely from camera driving data in an end-to-end manner, without relying on HD maps, and predicts multimodal future trajectories. We show that our model outperforms previous prediction baselines on the NuScenes and Lyft datasets. The code and trained models are available at https://github.com/wayveai/fiery.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/02357239ad190b183e3ac20292af55328dde1257

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Planning-oriented Autonomous Driving (2022, methodology): “%) compared to FIERY [35] and BEVerse [105] with heavy augmentations, respectively.”
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022, background): “Compared to simply stacking BEV in [18, 38, 6], our temporal self-attention can more effectively model long temporal dependency.”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, background): “The distance is calculated at several fixed positions along the Y-axis: [5, 10, 15, 20, 30, 40, 50, 60, 80, 100] for 3D anchor lanes.”
OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “Considering the dimension gap between the 2D image input and the 3D prediction, recent studies for vision-based 3D perception first construct the BEV feature representations and then perform various downstream tasks on the BEV space [19, 28, 30, 37, 58, 38, 60, 40, 18, 1, 42].”
ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning (2022, methodology): “The common practice includes generating uncertainty from data distribution for current state and feeding it in a temporal model to infer predictions under a window of future horizons [25, 47].”
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, methodology): “We random drop n ∈ [1, 4] cameras of the ego vehicle, and demonstrate the performance decrease for both SinBEVT (no collaboration) and CoBEVT in Fig.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, background): “To achieve this, the development of a robust model of the world is imperative [1] and huge efforts have been made in the past to build such predictive world models for autonomous driving [2, 3, 4, 5, 6].”
Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, neutral): “More recently, vision-based 3D object detection has gained more attention due to its low cost and rich semantic content [41, 46, 48, 22, 26, 33, 28, 13, 27, 15, 21, 31, 25].”
VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023, background): “3.4).”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, positive): “This procedure can be performed in conjunction with distribution-based planners [53, 148] that can make action proposals to reduce the searching space.”
BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision (2022, background): “ing the view transformation module [17, 28, 43] and incorporating an ever-growing list of downstream tasks [9, 28] into the new recognition framework, the study of image backbones in BEV models receives far less attention.”
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, background): “camera images as inputs, it has demonstrated competitive performance in various 3D perception tasks including depth estimation [18, 57], 3D object detection [29, 28, 35, 33, 21], and semantic map construction [65, 20, 1].”
Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction (2023, background): “Though lacking direct sensing of depth information, vision-based models empowered by surrounding cameras demonstrate promising performance on various 3D perception tasks such as depth estimation [18,45], semantic map reconstruction [1,20,51], and 3D object detection [28,31,49].”
VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, background): “However, recent developments in end-to-end motion prediction methods [17, 53, 14, 22] perform perception and motion prediction jointly.”