Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction

By Bo Yang, Sen Wang, Andrew Markham, Niki Trigoni. 2018

156 citations recorded by the source corpus.

We study the problem of recovering an underlying 3D shape from a set of images. Existing learning based approaches usually resort to recurrent neural nets, e.g., GRU, or intuitive pooling operations, e.g., max/mean poolings, to fuse multiple deep features encoded from input images. However, GRU based approaches are unable to consistently estimate 3D shapes given different permutations of the same set of input images as the recurrent unit is permutation variant. It is also unlikely to refine the 3D shape given more images due to the long-term memory loss of GRU. Commonly used pooling approaches are limited to capturing partial information, e.g., max/mean values, ignoring other valuable features. In this paper, we present a new feed-forward neural module, named AttSets, together with a dedicated training algorithm, named FASet, to attentively aggregate an arbitrarily sized deep feature set for multi-view 3D reconstruction. The AttSets module is permutation invariant, computationally efficient and flexible to implement, while the FASet algorithm enables the AttSets based network to be remarkably robust and generalize to an arbitrary number of input images. We thoroughly evaluate FASet and the properties of AttSets on multiple large public datasets. Extensive experiments show that AttSets together with FASet algorithm significantly outperforms existing aggregation approaches.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep multi-view learning methods: A review (2021, methodology): “Yang et al. [136] propose a joint framework for discriminative embedding and spectral clustering in a dual455 auto-encoder network.”
Multi-view 3D Reconstruction with Transformers (2021, methodology): “, 4), the accuracy of methods will be saturated, showing the difficulty of learning complementary knowledge from a large set of independent CNN feature extraction units [28, 27].”
RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (2019, background): “In particular, inspired by [65], our attentive pooling unit consists of the following steps.”
On the Limitations of Representing Functions on Sets (2019, methodology): “Yang et al. (2018) (Dinh et al., 2016) as a bijective function which sequentially computes the parameters of a Student-t process.”