Dual PatchNorm

By Manoj Kumar, Mostafa Dehghani, Neil Houlsby. 2023

14 citations recorded by the source corpus.

We propose Dual PatchNorm: two Layer Normalization layers (LayerNorms), before and after the patch embedding layer in Vision Transformers. We demonstrate that Dual PatchNorm outperforms the result of exhaustive search for alternative LayerNorm placement strategies in the Transformer block itself. In our experiments, incorporating this trivial modification, often leads to improved accuracy over well-tuned Vision Transformers and never hurts.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Ali Farhadi
2. Luke S. Zettlemoyer
3. Luc Van Gool
4. Mitchell Wortsman
5. Trevor Darrell
6. Ming-Hsuan Yang
7. Ludwig Schmidt
8. Dacheng Tao

In Electrical and Electronic Engineering

1. Tim Dettmers
2. Luke S. Zettlemoyer
3. Ali Farhadi
4. Ludwig Schmidt
5. Dan Alistarh
6. Elias Frantar

In Optimization & Training

1. Luke S. Zettlemoyer
2. Tim Dettmers
3. Ali Farhadi
4. Ludwig Schmidt
5. Ari S. Morcos
6. Mitchell Wortsman
7. Torsten Hoefler
8. Tal Ben-Nun

Citations in the corpus

• Stable and low-precision training for large-scale vision-language models (2023, methodology): “We initially tried adding a layer-norm before the patch embedding as in [34], but removed this as we found it to hurt performance at CLIP ViT-Huge scale.”