Pix2seq: A Language Modeling Framework for Object Detection

By Ting-Li Chen, Saurabh Saxena, Lala Li, David J. Fleet, Geoffrey E. Hinton. 2021

455 citations recorded by the source corpus.

We present Pix2Seq, a simple and generic framework for object detection. Unlike existing approaches that explicitly integrate prior knowledge about the task, we cast object detection as a language modeling task conditioned on the observed pixel inputs. Object descriptions (e.g., bounding boxes and class labels) are expressed as sequences of discrete tokens, and we train a neural network to perceive the image and generate the desired sequence. Our approach is based mainly on the intuition that if a neural network knows about where and what the objects are, we just need to teach it how to read them out. Beyond the use of task-specific data augmentations, our approach makes minimal assumptions about the task, yet it achieves competitive results on the challenging COCO dataset, compared to highly specialized and well optimized detection algorithms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

PaLM-E: An Embodied Multimodal Language Model (2023, background): “…VLMs are capable of simultaneously understanding both images and text, and can be applied to tasks such as visual question answering (Zhou et al., 2020; Zellers et al., 2021b), cap-tioning (Hu et al., 2022), optical character recognition (Li et al., 2021), and object detection (Chen et al., 2021b).”
Transformers in Vision: A Survey (2021, background): “Pix2Seq [35] is a generic Transformer-based framework, without any specialized task-specific modules, and learns to directly produce a sequence of tokens with object descriptions (bounding-boxes and class-labels).”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “Following [64], we represent objects as a sequence of discrete tokens.”
A Generalist Agent (2022, methodology): “Pix2Seq (Chen et al., 2022) also uses an LM-based architecture for object detection.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “• Object detection is a generative object-detection task inspired by Chen et al. (2021; 2022).”
UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes (2022, background): “There are papers that demonstrate that the NLP-like approach, based on autoregressive sequence models, is viable for some image tasks [4, 38].”
Kosmos-2: Grounding Multimodal Large Language Models to the World (2023, methodology): “OFA (Wang et al., 2022c) targets unifying various cross and mono modal tasks including image generation, visual grounding, image captioning, image classification, language modeling, etc., and has achieved impressive performances, by utilizing the concept of bounding boxes as tokens from Pix2Seq (Che”
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (2023, unknown): “Such metric design eases the future extension to more question types, such as box localization [12, 74, 70].”
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic (2023, methodology): “An interesting genre is Pix2seq (Chen et al., 2021), which formalizes the detection task as a sequence generation task.”
PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “Object detection is framed similarly to Pix2seq [72], with two key differences: the use of a natural language vocabulary, and class-conditioning.”
GPT-Driver: Learning to Drive with GPT (2023, methodology): “We note that language modeling has been employed in other tasks of computer vision and robotics, such as object detection Chen et al. (2021); Xue et al. (2022); Wang et al. (2023) and robotic control Brohan et al. (2023).”
Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking (2022, background): “On the other hand, transformers have recently shown an excellent capability to simplify frameworks for computer vision tasks, like object detection [6] and object segmentation [50].”
Autoregressive Visual Tracking (2023, methodology): “The first step in this framework is to construct discrete token sequences from object trajectories using a quantization and serialization scheme [8].”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “For example, iGibson (Shen et al., 2020; Li et al., 2021; Srivastava et al., 2021; Li et al., 2022a) simulates interactive household scenarios.”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “In this work, we follow [7, 10] in using ResNet [22] followed by trans-former encoder layers [55] as the feature extractor.”
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks (2023, unknown): “With the same backbone i.e. ResNet-50, our method outperforms Pix2Seq [7] by 1 .”
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, methodology): “Natural image understanding Pix2Seq uses the image-to-text architecture for core vision tasks such as object detection and instance segmentation (Chen et al., 2022b; 2021).”
DN-DETR: Accelerate DETR Training by Introducing Query DeNoising (2022, methodology): “[2] augments their sequence with synthetic noise objects, but is totally different from our method.”