Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net

By Wenjie Luo, Binh Yang, Raquel Urtasun. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

687 citations recorded by the source corpus.

In this paper we propose a novel deep neural network that is able to jointly reason about 3D detection, tracking and motion forecasting given data captured by a 3D sensor. By jointly reasoning about these tasks, our holistic approach is more robust to occlusion as well as sparse data at range. Our approach performs 3D convolutions across space and time over a bird's eye view representation of the 3D world, which is very efficient in terms of both memory and computation. Our experiments on a new very large scale dataset captured in several north american cities, show that we can outperform the state-of-the-art by a large margin. Importantly, by sharing computation we can perform all tasks in as little as 30 ms.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9bdc71eacf7440bab8d2852f229da537498c9546

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

GRIP: Graph-based Interaction-aware Trajectory Prediction (2019, background): “proposed a convolutional network for fast object detection, tracking and motion forecasting in [8].”
A survey of deep learning techniques for autonomous driving (2019, methodology): “tures are used to detect objects as 2D regions of interest [36] [37] [38] [39] [40] [41] or pixel-wise segmented areas in images [42] [43] [44] [45], 3D bounding boxes in LiDAR point clouds [33] [46] [47], as well as 3D representations of objects in combined cameraLiDAR data [48] [49] [34]. Examples”
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022, background): “For the human visual perception system, temporal information plays a crucial role in inferring the motion state of objects and identifying occluded objects, and many works in vision fields have demonstrated the effectiveness of using video data [2, 27, 26, 33, 19].”
Joint Object Detection and Multi-Object Tracking with Graph Neural Networks (2021, background): “Object detection [1]–[8] and data association [9]–[16] are two components of multi-object tracking (MOT), which is essential to perception in robotic systems such as autonomous driving [17]–[21].”
Argoverse: 3D Tracking and Forecasting With Rich Maps (2019, background): “[24] by discretizing the prediction space and attempting to predict one of eight common driving maneuvers.”
HDNET: Exploiting HD Maps for 3D Object Detection (2018, methodology): “78% AP within 0-70 meters, outperforming FAF [12] and PIXOR [3] by 4.”
CoverNet: Multimodal Behavior Prediction Using Trajectory Sets (2019, background): “Unimodal approaches output a single trajectory per agent [27, 6, 15, 1].”
Stereo R-CNN Based 3D Object Detection for Autonomous Driving (2019, methodology): “[6, 26, 15, 20, 31] utilize structured voxel grid representation to quantize the raw point cloud data, then use either 2D or 3D CNN to detect 3D object, while[20] takes multiple frames as input and generates 3D detection, tracking and motion forecasting simultaneously.”
ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning (2022, methodology): “Taking raw sensor data as input, end-to-end methods which focus on future trajectory prediction or take it as an intermediate step usually rely on LiDAR and HD map [34, 52, 51, 16, 48] to detect and predict.”
Learning Distilled Collaboration Graph for Multi-Agent Perception (2021, neutral): “, 2D/3D object detection [18, 27], tracking [23, 22] and segmentation [25, 16], etc.”
EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection (2020, background): “troduction The last decade has witnessed signicant progress in the 3D object detection task via dierent types of sensors, such as monocular images [1,36], stereo cameras [2], and LiDAR point clouds [43,22,39]. Camera images usually contain plenty of semantic features (e.g., color, texture) while sue”
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving (2021, background): “object detection [22, 12, 66, 7, 44, 31, 34, 61, 33, 37] and motion forecasting [36, 5, 35, 63, 6, 19, 38].”
Segment as Points for Efficient Online Multi-Object Tracking and Segmentation (2020, background): “tect objects of interests and then link objects into trajectories via data association. The data association can be accomplished on either the 2D image plane [4,6,7,14,33,42,38] or the 3D world space [1,8,10,20,25,39]. ATOM [7] introduces a novel tracking architecture, which consists of dedicated ta”
Accurate Monocular 3D Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving (2019, background): “LiDAR provides reliable depth information that can be used to accurately localize objects and characterize their shapes, many approaches [14, 19, 21, 27, 5, 33, 40] use LiDAR point clouds as their input, and get impressive detection results in autonomous driving scenarios.”
IntentNet: Learning to Predict Intention from Raw Sensor Data (2018, methodology): “This simple tracking system proposed in FaF [4] also allows us to recover missing false negatives and discard false positives by updating the classification scores based on previous predictions.”
Planning-oriented Autonomous Driving (2022, background): “Pioneering work FaF [66] predicts boxes in the future and aggregates past information to produce tracklets.”