Multi-view 3D Reconstruction with Transformers

By Dan Wang, Xinrui Cui, Xun Chen, Zhengxia Zou, Tianyang Shi, Septimiu E. Salcudean, Z. J. Wang, Rabab Kreidieh Ward. 2021

115 citations recorded by the source corpus.

Deep CNN-based methods have so far achieved the state of the art results in multi-view 3D object reconstruction. Despite the considerable progress, the two core modules of these methods - view feature extraction and multi-view fusion, are usually investigated separately, and the relations among multiple input views are rarely explored. Inspired by the recent great success in Transformer models, we reformulate the multi-view 3D reconstruction as a sequence-to-sequence prediction problem and propose a framework named 3D Volume Transformer. Unlike previous CNN-based methods using a separate design, we unify the feature extraction and view fusion in a single Transformer network. A natural advantage of our design lies in the exploration of view-to-view relationships using self-attention among multiple unordered inputs. On ShapeNet - a large-scale 3D reconstruction benchmark, our method achieves a new state-of-the-art accuracy in multi-view reconstruction with fewer parameters (70% less) than CNN-based methods. Experimental results also suggest the strong scaling capability of our method. Our code will be made publicly available.

Subfields: Computer Vision and Pattern Recognition, Model Architectures, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Jiang Tian
  2. Feng Hou
  3. Yixin Wang
  4. Yao Zhang
  5. Jin Yuan
  6. Jianping Fan
  7. Yang Liu
  8. Zhongchao Shi

In Model Architectures

  1. Jiang Tian
  2. Feng Hou
  3. Yixin Wang
  4. Yao Zhang
  5. Jin Yuan
  6. Jianping Fan
  7. Yang Liu
  8. Zhongchao Shi

Citations in the corpus