Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks

By Chunjie Luo, Jianfeng Zhan, Lei Wang, Qiang Yang. 2017

246 citations recorded by the source corpus.

Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change of input distribution, thus results in poor generalization, and aggravates the internal covariate shift which slows down the training. To bound dot product and decrease the variance, we propose to use cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot product in neural networks, which we call cosine normalization. We compare cosine normalization with batch, weight and layer normalization in fully-connected neural networks, convolutional networks on the data sets of MNIST, 20NEWS GROUP, CIFAR-10/100, SVHN. Experiments show that cosine normalization achieves better performance than other normalization techniques.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Optimization & Training

Citations in the corpus

Task-Aware Information Routing from Common Representation Space in Lifelong Learning (2023, methodology): “As cosine similarity is widely used in high-dimensional spaces (Luo et al., 2018), we plot the cosine similarity between respective feature extractor and selector weight matrices of each TAMs in Figure 6.”
An EM Framework for Online Incremental Learning of Semantic Segmentation (2021, positive): “It is worth noting that we are the first to adopt cosine normalization in the online class incremental learning problem, even though it has been used in other problems [22, 26, 35].”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, methodology): “Note that this combination at the last layer is equivalent to cosine normalization [22] with a learned scale.”
Essentials for Class Incremental Learning (2021, background): “It consists of a (1) cosine-normalized layer (cos) [8, 18, 12], where the features and class-weight vectors in the final layer are normalized to lie in a high-dimensional sphere.”
Insights from the Future for Continual Learning (2020, background): “…architecture with a convolutional feature extractor h = f ( x ) (where x is the input image and h is the feature vector) and a cosine classiﬁer clf [29, 17], a fully-connected layer with the dot-product replaced by a cosine similarity: Remark that θ c , the parameter vector for the c -th class in…”
PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning (2020, methodology): “We also compare the results with the Cosine classiﬁer [25,13] against the Local Similarity Classiﬁer (LSC) with NCA loss.”
TaskNorm: Rethinking Batch Normalization for Meta-Learning (2020, background): “There exist additional NLs including Weight Normalization (Salimans & Kingma, 2016), Cosine Normalization (Luo et al., 2018), Filter Response Normalization (Singh & Krishnan, 2019), among many others.”
Analyzing and Improving the Training Dynamics of Diffusion Models (2023, unknown): “We rectify this by switching to cosine attention [13, 43, 45] that normalizes the vectors prior to computing the dot products.”
Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation (2021, methodology): “In practice, we replace the classifier by a cosine classifier [22], where the final inner product is discarded in favor of a cosine similarity.”
Learning a Unified Classifier Incrementally via Rebalancing (2019, methodology): “Although cosine normalization is widely adopted in other visual tasks [33, 10, 27, 25], it is first introduced for incremental learning here.”
Spot and Learn: A Maximum-Entropy Patch Sampler for Few-Shot Image Classification (2019, background): “Cosine similarity layers has been recently explored in [16] and has been applied to few-shot learning in [7, 19].”
Sentiment Classification Using Document Embeddings Trained with Cosine Similarity (2019, background): “In the context of fully-connected neural networks and convolutional neural networks, (Luo et al., 2017) uses cosine similarity instead of dot product in computing a layer’s pre-activation as a regularization mechanism.”
Low-Shot Learning with Imprinted Weights (2017, background): “Recent work in cosine normalization [9] discusses a similar idea of replacing the inner product with a cosine similarity for bounded activations and stable training, while we arrive at this design from a different direction.”
Incremental Concept Learning via Online Generative Memory Recall (2019, background): “However, training in this way will lead to large feature variance[15], [16], which will result in larger weights variation and harm the implicit memory during continual learning.”
Hyperspherical Prototype Networks (2019, unknown): “The potential of angular similarities has also been investigated in other layers of deep networks (Liu et al., 2017b; Luo et al., 2017).”
Flow-Guided Feature Aggregation for Video Object Detection (2017, methodology): “Here, we use the cosine similarity metric [27] to measure the similarity between the warped features and the features extracted from the reference frame.”