TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?

By Michael S. Ryoo, A. Piergiovanni, Anurag Arnab, Mostafa Dehghani, Anelia Angelova. 2021

181 citations recorded by the source corpus.

In this paper, we introduce a novel visual representation learning which relies on a handful of adaptively learned tokens, and which is applicable to both image and video understanding tasks. Instead of relying on hand-designed splitting strategies to obtain visual tokens and processing a large number of densely sampled patches for attention, our approach learns to mine important tokens in visual data. This results in efficiently and effectively finding a few important visual tokens and enables modeling of pairwise attention between such tokens, over a longer temporal horizon for videos, or the spatial content in images. Our experiments demonstrate strong performance on several challenging benchmarks for both image and video recognition tasks. Importantly, due to our tokens being adaptive, we accomplish competitive results at significantly reduced compute amount. We obtain comparable results to the state-of-the-arts on ImageNet while being computationally more efficient. We also confirm the effectiveness of the approach on multiple video datasets, including Kinetics-400, Kinetics-600, Charades, and AViD. The code is available at: https://github.com/google-research/scenic/tree/main/scenic/projects/token_learner

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Multiview Transformers for Video Recognition (2022, methodology): “When using additional JFT-300M pretraining, our “Huge” model outperforms other recent transformer models using the same pretraining dataset [3,55].”
Florence: A New Foundation Model for Computer Vision (2021, neutral): “, 2021b), TokenLearner (Ryoo et al., 2021), on Kinetics-400 and Kinetics-600.”
PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “We investigate state estimation vectors, Vision Transformers (ViTs) (Dosovitskiy et al., 2020; Chen et al., 2022; Ryoo et al., 2021) for 2D image features, and the 3D-aware Object Scene Representation Transformer (OSRT) (Sajjadi et al.”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, neutral): “9 TokenLearner [45] 256(8)(2)×64(64) 256(8)(2)×64(64) 4×3 85.”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “Inspired by the success of Transformers in NLP [20,76], the Vision Transformer (ViT) [22] was recently proposed and has been successfully applied to multiple computer vision tasks, including image classification [44, 72, 73, 86], object detection [44, 79, 88], semantic segmentation [44, 82, 91] and ”
Masked Feature Prediction for Self-Supervised Visual Pre-Training (2021, unknown): “Still, 300 epochs of MaskFeat pre-training improve the scratch MViT-S, 16 × 4 [56] Sup., JFT-300M 84.9 95.8 3981 × 3 × 4 TokenLearner [75] Sup., JFT-300M 85.4 N/A 4076 × 3 × 4 Florence ↑ 384 [95] Text, FLD-900M 86.5 3.”
Video-Text Pre-training with Learned Regions (2021, methodology): “Specifically, inspired by [59], we extract K region features with complete semantics in space from the sequential features F ′ .”
DaViT: Dual Attention Vision Transformers (2022, background): “Similarly, follow-up works [25,49,56,47,59,20,73,43,44] model global relationships on the patch-level tokens.”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Inspired by the success of Transformers in Natural Language Processing (NLP), plenty of works [160], [167] employed Transformers for HAR from videos.”
From Sparse to Soft Mixtures of Experts (2023, methodology): “Many existing works merge , mix or fuse input tokens to reduce the input sequence length (Jaegle et al., 2021; Ryoo et al., 2021; Renggli et al., 2022; Wang et al., 2022), typically using attention-like weighted averages with fixed keys, to try to alleviate the quadratic cost of self-attention with…”
SCENIC: A JAX Library for Computer Vision Research and Beyond (2021, methodology): “BERT [11] Text Self-supervised pre-training, GLUE/SuperGLUE benchmark ViViT [6] Video Video classification TokenLearner [25] Image, Video Image/Video classification OmniNet [28] Image Image classification PolyViT [19] Image, Audio, Video Image/Audio/Video classification”