You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection

By Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, Wenyu Liu. 2021

416 citations recorded by the source corpus.

Can Transformer perform 2D object- and region-level recognition from a pure sequence-to-sequence perspective with minimal knowledge about the 2D spatial structure? To answer this question, we present You Only Look at One Sequence (YOLOS), a series of object detection models based on the vanilla Vision Transformer with the fewest possible modifications, region priors, as well as inductive biases of the target task. We find that YOLOS pre-trained on the mid-sized ImageNet-1k dataset only can already achieve quite competitive performance on the challenging COCO object detection benchmark, e.g., YOLOS-Base directly adopted from BERT-Base architecture can obtain 42.0 box AP on COCO val. We also discuss the impacts as well as limitations of current pre-train schemes and model scaling strategies for Transformer in vision through YOLOS. Code and pre-trained models are available at https://github.com/hustvl/YOLOS.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Transformers in Vision: A Survey (2021, background): “You Only Look at One Sequence (YOLOS) [62] is a simple, attention-only architecture directly built upon the ViT [57, 201].”
Simple Open-Vocabulary Object Detection with Vision Transformers (2022, methodology): “Compared to [9], which uses additional “detection” tokens, we further simplify the model by predicting one object instance directly from each image token.”
Inception Transformer (2022, neutral): “, object detection [10, 11] and segmentation [12, 13].”
Improving Diffusion Model Efficiency Through Patching (2022, background): “Patching also sees widespread usage in vision transformers [6], which have seen success on various other image tasks [8; 33].”
MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction (2022, background): “MapTR parallelly infers a ﬁxed-size set of N map elements in a single pass, following the end-to-end paradigm of DETR (Carion et al., 2020; Fang et al., 2021).”
LocalMamba: Visual State Space Model with Windowed Selective Scan (2024, neutral): “…the untapped potential of Transformers in visual tasks, sparking a surge of research aimed at refining their architecture design [45] and training methodologies [18, 31, 46, 47, 53], boosting computational efficiency [3,22,33,49], and extending their application scope [8, 25, 26, 38, 44, 56, 58].”
Vision Transformers Need Registers (2023, background): “However, most approaches add new tokens either to provide the network with new information as for example [SEP] tokens in BERT, provide opportunity to spend more computation on the input as seen with the tape tokens in AdaTape (Xue et al., 2023), or to gather information in these tokens, and use the”
A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS (2023, methodology): “One of the first attempts at using transformers for object detection was You Only Look at One Sequence or YOLOS [128], turned a pre-trained Vision Transfomer (ViT) [129] from image classification to object detection, achieving 42.0 % AP on MS COCO dataset.”
HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation (2022, neutral): “Vision transformers have also shown impressive results in other vision tasks, including [61, 21], which offer end-to-end transformer-based models for object detection, and [46, 26] for semantic and instance segmentation.”
VLP: A Survey on Vision-language Pre-training (2022, background): “In addition, there are some image-related downstream tasks for evaluating the image-text pre-training models, including semantic segmentation[101, 102], and object detection[103, 104].”
A Survey of Visual Transformers (2021, methodology): “Fang et al. [88] presented an encoder-only decoder YOLOS, a pure sequence-to-sequence Transformer to unify the classiﬁcation and detection tasks.”
A Survey on Vision Transformer (2020, background): “[64] explored how to transfer the pure ViT structure that is pre-trained on ImageNet to the more challenging object”
ByteTrack: Multi-Object Tracking by Associating Every Detection Box (2021, methodology): “With the rapid development of object detection [51,27,50,36,11,24,59,57,21,22,63], more and more methods begin to utilize more powerful detectors to obtain higher tracking performance.”