Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture

By Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, Derek Hoiem. 2021

58 citations recorded by the source corpus.

Computer vision systems today are primarily N-purpose systems, designed and trained for a predefined set of tasks. Adapting such systems to new tasks is challenging and often requires nontrivial modifications to the network architecture (e.g. adding new output heads) or training process (e.g. adding new losses). To reduce the time and expertise required to develop new applications, we would like to create general purpose vision systems that can learn and perform a range of tasks without any modification to the architecture or learning process. In this paper, we propose GPV-1, a task-agnostic vision-language architecture that can learn and perform tasks that involve receiving an image and producing text and/or bounding boxes, including classification, localization, visual question answering, captioning, and more. We also propose evaluations of generality of architecture, skill-concept11For this work, we define concepts, skills and tasks as follows: Concepts - nouns (e.g. car, person, dog), Skills - operations that we wish to perform on the given inputs (e.g. classification, object detection, image captioning), Tasks - predefined combinations of a set of skills performed on a set of concepts (e.g. ImageNet classification task involves the skill of image classification across 1000 concepts). transfer, and learning efficiency that may informfuture work on general purpose vision. Our experiments indicate GPV-1 is effective at multiple tasks, reuses some concept knowledge across tasks, can perform the Referring Expressions task zero-shot, and further improves upon the zero-shot performance using a few training samples.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, neutral): “Similar to ours, recent works study the impact of language instructions in modifying a model behavior (Hase and Bansal, 2021; Ye and Ren, 2021; Gupta et al., 2021; Zhong et al., 2021).”
Generalized Decoding for Pixel, Image, and Language (2022, background): “Furthermore, X-Decoder also beats other general-purpose models like UniT [32], GPV [27], UniTAB [85] and Unified-IO [54].”
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling (2021, methodology): “Instead of extra alignment score prediction [23,78,80,45], we represent word-box alignments inline with two introduced special tokens <obj> and <\obj>.”
Webly Supervised Concept Expansion for General Purpose Vision Models (2022, methodology): “We train GPV-1 to 40 epochs following [25](3).”
FindIt: Generalized Localization with Natural Language Queries (2022, methodology): “Last but not least, we discover that a standard object detector and detection losses [69] are sufficient and surprisingly effective for REC, LOC, and DET tasks without a need for task-specific design and losses [13,21,51,55,88,89,91].”
Satlas: A Large-Scale, Multi-Task Dataset for Remote Sensing Image Understanding (2022, methodology): “SATNET is inspired by recent work that employ task-specific output heads [20, 27, 32], as well as methods that synthesize features across remote sensing image time series [21,25].”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, methodology): “The General Robust Image Task (GRIT) Benchmark (Gupta et al., 2022b) is an evaluation-only benchmark designed to measure the performance of models across multiple tasks, concepts, and data sources.”