In defense of soft-assignment coding

By Lingqiao Liu, Lei Wang, Xinwang Liu. 2011

492 citations recorded by the source corpus.

In object recognition, soft-assignment coding enjoys computational efficiency and conceptual simplicity. However, its classification performance is inferior to the newly developed sparse or local coding schemes. It would be highly desirable if its classification performance could become comparable to the state-of-the-art, leading to a coding scheme which perfectly combines computational efficiency and classification performance. To achieve this, we revisit soft-assignment coding from two key aspects: classification performance and probabilistic interpretation. For the first aspect, we argue that the inferiority of soft-assignment coding is due to its neglect of the underlying manifold structure of local features. To remedy this, we propose a simple modification to localize the soft-assignment coding, which surprisingly achieves comparable or even better performance than existing sparse or local coding schemes while maintaining its computational advantage. For the second aspect, based on our probabilistic interpretation of the soft-assignment coding, we give a probabilistic explanation to the magic max-pooling operation, which has successfully been used by sparse or local coding schemes but still poorly understood. This probability explanation motivates us to develop a new mix-order max-pooling operation which further improves the classification performance of the proposed coding scheme. As experimentally demonstrated, the localized soft-assignment coding achieves the state-of-the-art classification performance with the highest computational efficiency among the existing coding schemes.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation (2015, neutral): “A soft codebook mapping like in [15, 40] fits trivially into the framework mðdÞðlÞ 1⁄4 exp ð bdðl; bdÞÞ P ðdÞ exp ð bdðl; bdÞÞ ð36Þ Pixel-Wise Explanations of Non-Linear Whole Image Classifier Decisions”
Bag of visual words and fusion methods for action recognition: Comprehensive study and good practice (2014, neutral): “In this paper, we have comprehensively studied each step in the BoVW pipeline and tried to uncover good practice to build a more accurate and efficient action recognition system.”
Action Recognition with Stacked Fisher Vectors (2014, methodology): “Recently, advanced feature encoding methods have been introduced for action recognition, such as soft-assignment [21,36], vector of locally aggregated descriptors [9,8], and Fisher coding [24,36,32].”
A Review on Human Activity Recognition Using Vision-Based Method (2017, background): “(2011) [71], 398 Salient coding (SC) Huang et al.”
Deep filter banks for texture recognition and segmentation (2015, methodology): “Texton distributions were the early versions of “bag-of-words” representations, a dominant approach in recognition in the early 2000s, since then improved by new pooling schemes such as soft-assignment [27, 42, 48] and Fisher Vectors (FVs) [32].”
Enhanced Performance of Brain Tumor Classification via Tumor Region Augmentation and Partition (2015, methodology): “Other more sophisticated methods may also be used, such as SC, soft assignment (SA) [24,25], orthogonal matching pursuit (OMP), locality-constrained linear coding [26], and local anchor embedding [27].”
Unsupervised feature learning framework for no-reference image quality assessment (2012, methodology): “In image classification problems, however, max-pooling has demonstrated higher performance in image classification than average-pooling (or sum-pooling) [10, 28].”
Deep convolutional filter banks for texture recognition and segmentation (2014, methodology): “Texton distributions were the early versions of “bag-of-words” representations, a dominant approach in recognition in the early 2000s, since then improved by new pooling schemes such as soft-assignment [26, 39, 45] and Fisher Vectors (FVs) [31].”
Human Action Recognition by Mining Discriminative Segment with Novel Skeleton Joint Feature (2013, methodology): “a segment of frames), we utilize “max pooling” operation, which has shown its significance for robust representation in image classification [14–16], to get its representation from the features of frames.”
Blind Image Quality Assessment Based on High Order Statistics Aggregation (2016, methodology): “The representative codebook-based framework (a.k.a bag of words model) [30] and several extensions [31], [32] which have gained popularity in image classification and retrieval tasks, have also been employed for BIQA.”
A Comparative Study of Encoding, Pooling and Normalization Methods for Action Recognition (2012, methodology): “In addition to VQ, several effective encoding methods have been proposed, such as: Fisher Kernel Encoding [4], Soft-assignment Encoding [15,1], Sparse Encoding [2], Localityconstrained Linear Coding [3].”
Deep Filter Banks for Texture Recognition, Description, and Segmentation (2015, methodology): “Several related methods used concepts from sparse coding to deﬁne the local descriptor encoder (Zhou et al. 2010; Liu et al. 2011).”
Salient Color Names for Person Re-identification (2014, background): “Inspired by the idea of saliency [8] which is also reflected in other classic coding strategies (e.g. locality-constrained linear coding (LLC) [24], salient coding (SC) [8], local manifold-constrained coding (LMC) [29] and localized soft-assignment coding (LSC) [16]) in image classification, we assi”
From BoW to CNN: Two Decades of Texture Representation for Texture Classification (2018, methodology): “Another noticeable pooling method is the mix-order max pooling method which considers the information of visual word occurrence frequency [127].”
Deep learning for visual understanding (2017, unknown): “To avoid using unreliable likelihood, we further adopt the idea of localized soft assignment [243] and only keep the assignment strengths with large values, and modify Eq 4.”
Hallucinating IDT Descriptors and I3D Optical Flow Features for Action Recognition With CNNs (2019, methodology): “Similar models such as Soft Assignment (SA) [62, 33] and Localized Soft Assignment (LcSA) [45, 38] use the Component Membership Probability (CMP) of GMM to assign each descriptor with some probability to visual words followed by average or non-linear pooling [38, 70].”