SORT: Second-Order Response Transform for Visual Recognition

By Yan Wang, Lingxi Xie, Chenxi Liu, Ya Zhang, Wenjun Zhang, Alan L. Yuille. 2017

56 citations recorded by the source corpus.

In this paper, we reveal the importance and benefits of introducing second-order operations into deep neural networks. We propose a novel approach named Second-Order Response Transform (SORT), which appends element-wise product transform to the linear sum of a two-branch network module. A direct advantage of SORT is to facilitate cross-branch response propagation, so that each branch can update its weights based on the current status of the other branch. Moreover, SORT augments the family of transform operations and increases the nonlinearity of the network, making it possible to learn flexible functions to fit the complicated distribution of feature space. SORT can be applied to a wide range of network architectures, including a branched variant of a chain-styled network and a residual network, with very light-weighted modifications. We observe consistent accuracy gain on both small (CIFAR10, CIFAR100 and SVHN) and big (ILSVRC2012) datasets. In addition, SORT is very efficient, as the extra computation overhead is less than 5%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Few-Shot Image Recognition by Predicting Parameters from Activations (2017, methodology): “With datasets of this scale, machine learning methods that have large capacity start to prosper, and the most successful ones are convolutional neural network based [11, 12, 16, 28, 32].”
Deep High-Resolution Representation Learning for Visual Recognition (2019, background): “The possible reason might be that multiplication increases the training difﬁculty as pointed in [145].”
Single-Shot Object Detection with Enriched Semantics (2017, background): “With the emergence of deep neural networks, computer vision has been improved significantly in many aspects such as image classification [11, 13, 15, 24, 27], object detection [3, 17, 20, 21, 25], and segmentation [2, 10, 18].”
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (2022, methodology): “In our implementation of gnConv, the higher-order spatial interactions are based on the gating mechanism, which has also been investigated in LSTM [23] and some vision modules [55].”
Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students (2018, unknown): “80% top5) and Second-Order Response Transform (SORT) [32] (0.”
Bounding Box Regression With Uncertainty for Accurate Object Detection (2018, unknown): “aining From our early experiments, we observe that training from scratch with KL Loss is unstable. We hypothesize that the instability is introduced by the second order term 2(x g x e) =2 in our loss [54] (and x are both from fc7 feature and element-wisely multiplied). Therefore, we ﬁrst warm up the”
Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization (2018, methodology): “Instead of adding a geometric mean which causes training instability [57], we propose to use cross attentions as shown in Eq.”
Global Second-Order Pooling Convolutional Networks (2018, methodology): “In contrast, our networks are much better, achieving over 2.8% and 2.6% higher accuracies than FBN and SORT.”