LambdaNetworks: Modeling Long-Range Interactions Without Attention

By Irwan Bello. 2021

193 citations recorded by the source corpus.

We present lambda layers -- an alternative framework to self-attention -- for capturing long-range interactions between an input and structured contextual information (e.g. a pixel surrounded by other pixels). Lambda layers capture such interactions by transforming available contexts into linear functions, termed lambdas, and applying these linear functions to each input separately. Similar to linear attention, lambda layers bypass expensive attention maps, but in contrast, they model both content and position-based interactions which enables their application to large structured inputs such as images. The resulting neural network architectures, LambdaNetworks, significantly outperform their convolutional and attentional counterparts on ImageNet classification, COCO object detection and COCO instance segmentation, while being more computationally efficient. Additionally, we design LambdaResNets, a family of hybrid architectures across different scales, that considerably improves the speed-accuracy tradeoff of image classification models. LambdaResNets reach excellent accuracies on ImageNet while being 3.2 - 4.4x faster than the popular EfficientNets on modern machine learning accelerators. When training with an additional 130M pseudo-labeled images, LambdaResNets achieve up to a 9.5x speed-up over the corresponding EfficientNet checkpoints.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “LambdaNetworks [2], AFT [97] and FLASH [44] are other attempts at replacing attention in the context of image classiﬁcation and language modeling.”
VOLO: Vision Outlooker for Visual Recognition (2021, neutral): “Recent progress on image classification is mostly driven by attention-based models [71, 57, 26, 1] or specifically the transformer-based models.”
Training data-efficient image transformers & distillation through attention (2020, background): “Nevertheless hybrid architectures that combine convnets and transformers, including the self-attention mechanism, have exhibited competitive results in image classification[modif: (Bello et al., 2019; Bello, 2021; Wu et al., 2020)], detection (Carion et al.”
Perceiver: General Perception with Iterative Attention (2021, background): “A number of scalable self-attention variants have also been proposed (Katharopoulos et al., 2020; Peng et al., 2021; Child et al., 2019; Tay et al., 2021; 2020; Bello, 2021), but they may be insufficient by themselves to achieve good performance on high-dimensional perceptual tasks (Tay et al.”
High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “However we find that width scaling is ineffective for ResNet backbones, consistent with Bello (2021), who attain strong performance when only scaling depth and input resolution.”
Dynamic Neural Networks: A Survey (2021, methodology): “Rather than generating standard convolutional weights, LambdaNetworks [120] learns to predict the weights of linear projections based on the contexts of each pixel together”
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, methodology): “LambdaNetworks [2] uses the lambda layer, an efficient self-attention to replace the convolution in the CNN.”
MLP-Mixer: An all-MLP Architecture for Vision (2021, background): “Examples of improvements due to augmentation or regularization changes include Cubuk et al. [11], who attain excellent classiﬁcation performance with learned data augmentation, and Bello et al. [4], who show that canonical ResNets are still near state-of-the-art, if one uses recent training and augm”
Going deeper with Image Transformers (2021, methodology): “Lastly, LambdaNetworks [3] decomposes attention into an approximated content attention and a batch-amortized positional attention component.”
Multiscale Vision Transformers (2021, background): “MViT-B 16x4 MViT-B 32x2 [1] ViViT-L ImageNet-21K [6] TimeSformer ImageNet-21K [78] VTN ImageNet-1K / 21K”
Bottleneck Transformers for Visual Recognition (2021, background): “[1,0,0] is closer to the performance of BoT50 ([1,1,1]) compared to the other configurations, however a difference of 0.”
Nyströmformer: A Nystöm-based Algorithm for Approximating Self-Attention (2021, background): “The lambda layers in (Bello 2021), can also be thought of as an efficient relative attention mechanism.”
Scaling Local Self-Attention for Parameter Efficient Visual Backbones (2021, background): “Other forms of non-global pixel interaction can also be implemented efficiently [6, 18, 54, 1].”
EfficientNetV2: Smaller Models and Faster Training (2021, neutral): “We would like to highlight a few interesting observations: • Scaling up data size is more effective than simply scaling up model size in high-accuracy regime : when the top-1 accuracy is beyond 85%, it is very difﬁcult to further improve it by simply increasing model size (Radosavovic et al., 2020) ”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “Bello [45] proposes an approximated content attention with a positional attention component.”
XCiT: Cross-Covariance Image Transformers (2021, methodology): “Closer in spirit to our work, Lambda layers propose a way to ensure global interaction in ResNet models [4].”