Tangent Convolutions for Dense Prediction in 3D

By Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

612 citations recorded by the source corpus.

We present an approach to semantic scene analysis using deep convolutional networks. Our approach is based on tangent convolutions - a new construction for convolutional networks on 3D data. In contrast to volumetric approaches, our method operates directly on surface geometry. Crucially, the construction is applicable to unstructured point clouds and other noisy real-world data. We show that tangent convolutions can be evaluated efficiently on large-scale point clouds with millions of points. Using tangent convolutions, we design a deep fully-convolutional network for semantic segmentation of 3D point clouds, and apply it to challenging real-world datasets of indoor and outdoor 3D environments. Experimental results show that the presented approach outperforms other recent deep network constructions in detailed analysis of large 3D scenes.

Subfields: Computational Mechanics, Environmental Engineering, Geology

View Paper: https://www.semanticscholar.org/paper/b256e7155dd1a27ed9233aa4b47fd6334f5f243b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Environmental Engineering

In Geology

Citations in the corpus

ACNe: Attentive Context Normalization for Robust Permutation-Equivariant Learning (2020, background): “These include graph convolutional networks [13, 26], VoxelNets [61], tangent convolutions [44], and many others.”
Self-Supervised Pretraining of 3D Features on any Point-Cloud (2021, methodology): “Point-cloud based models [66, 68] are widely used in classification and segmentation tasks [6, 40, 45, 66, 68, 91, 98, 100, 106, 107], 3D reconstruction [23, 90, 110] and 3D object detection [63, 64, 77, 98, 112, 112, 114].”
Fast end-to-end learning on protein surfaces (2020, methodology): “based on 2D views [44], volumetric [38], space partitioning [35, 42, 39] and implicit representations [14]), geometric deep learning seeks to develop a non-Euclidean analogy of filtering and pooling operations.”
An End-to-End Transformer Model for 3D Object Detection (2021, methodology): “Projection methods [3, 19, 25, 26, 59, 60, 65] project 3D data into 2D planes and convert it into 2D grids.”
SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation (2020, methodology): “In comparison, PointNet-based methods [35,36,22,43,46] do not perform well in either accuracy and speed except RandLA-Net [15] which is a new efficient and effective work.”
PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies (2022, neutral): “[49, 21, 41, 40] project point clouds onto pseudo grids to allow for regular convolutions.”
GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping (2020, methodology): “After that, many methods [34, 38, 3, 23, 12, 43, 39, 40, 20, 19, 49, 47, 44, 46, 15] are proposed to perform point cloud classification and segmentation.”
Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion (2020, background): “Previous works projected all points scanned by the rotating LiDAR onto 2D images by plane projection (Lawin et al. 2017; Boulch, Le Saux, and Audebert 2017; Tatarchenko et al. 2018) or spherical projection (Wu et al. 2018, 2019).”
3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans (2018, methodology): “With the availability of 3D shape databases [36, 3, 32] and annotated RGB-D datasets [29, 1, 5, 2], these network architectures are now being used for 3D object classification [36, 20, 24, 28], semantic segmentation [5, 34, 6], and object or scene completion [8, 32, 9].”
ShellNet: Efficient Point Cloud Convolutional Neural Networks Using Concentric Shells Statistics (2019, background): “[33] perform convolution on the local tangent planes.”
PointConv: Deep Convolutional Networks on 3D Point Clouds (2018, methodology): “We compare our algorithm with Tangent Convolutions [37], SPLAT Net [35], PointNet++ [28] and ScanNet [5].”
RangeNet ++: Fast and Accurate LiDAR Semantic Segmentation (2019, methodology): “Leveraging large datasets for other contexts [5], [8], several deep learning-based methods for 3D semantic segmentation were recently developed, such as PointNet [14], PointNet++ [15], TangentConvolutions [20], SPLATNet [19], SuperPointGraph [10], and SqueezeSeg [21], [22].”
PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors (2018, background): “propose convolutions in the tangent space as a way of operating on the local 2D projection [39].”
RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (2019, methodology): “It can be seen that our RandLA-Net surpasses all point based approaches [43, 26, 49, 44, 51] by a large margin.”
4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks (2019, background): “Currently, there are various concurrent works for 3D perception: a dense 3D convolution [5], pointnet-variants [23, 24], continuous convolutions [12, 16], surface convolutions [21, 30], and an octree convolution [25].”
PointWeb: Enhancing Local Neighborhood Features for Point Cloud Processing (2019, methodology): “Compared approaches are 3DCNN [2], PointNet [18], TCDP [27], PointNet++ [20], PointCNN [13], and our PointWeb.”
PCT: Point cloud transformer (2020, background): “[23] proposed tangent convolution, which can learn surface geometric features from projected virtual tangent images.”
3D Point Capsule Networks (2018, background): “It is also common to process point sets by using local projections reducing the convolution operation down to two dimensions [32, 13].”
PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds (2021, background): “[45] propose to map local surface points onto a tangent plane and further uses 2D convolutional operators, and FPConv [25] flattens local patches onto regular 2D grids with soft weights.”
MeshCNN: a network with an edge (2019, methodology): “Tatarchenko et al. [2018] introduce tangent convolution, where a small neighborhood around each point is used to reconstruct the local function upon which convolution is applied.”