Vision Transformers are Good Mask Auto-Labelers

By Shiyi Lan, Xitong Yang, Zhiding Yu, Zuxuan Wu, José Manuel Álvarez, Anima Anandkumar. 2023

24 citations recorded by the source corpus.

We propose Mask Auto-Labeler (MAL), a high-quality Transformer-based mask auto-labeling framework for instance segmentation using only box annotations. MAL takes box-cropped images as inputs and conditionally generates their mask pseudo-labels. We show that Vision Transformers are good mask auto-labelers. Our method significantly reduces the gap between auto-labeling and human annotation regarding mask quality. Instance segmentation models trained using the MAL-generated masks can nearly match the performance of their fully-supervised counterparts, retaining up to 97.4% performance of fully supervised models. The best model achieves 44.1% mAP on COCO instance segmentation (test-dev 2017), outperforming state-of-the-art box-supervised methods by significant margins. Qualitative results indicate that masks produced by MAL are, in some cases, even better than human annotations.

Subfields: Civil and Structural Engineering, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Civil and Structural Engineering

1. Lei Zhang

In Computer Vision and Pattern Recognition

1. Jianwei Yang
2. Jianfeng Gao
3. Lei Zhang
4. Luc Van Gool
5. Feng Li
6. Hao Zhang
7. Dacheng Tao
8. Xiaogang Wang

In Representation & Self-Supervised Learning

1. Jianfeng Gao
2. Jianwei Yang
3. Siyi Liu
4. Lei Zhang
5. Philipp Koehn

Citations in the corpus

• A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Prominent methods de-sign teacher models or weak supervision loss, like Box-Inst [43], Box2Mask [31], DiscoBox [25] and Mask Auto-Labelers [24].”