Score Function Features for Discriminative Learning: Matrix and Tensor Framework

By Majid Janzamin, Hanie Sedghi, Anima Anandkumar. 2014

45 citations recorded by the source corpus.

Feature learning forms the cornerstone for tackling challenging learning problems in domains such as speech, computer vision and natural language processing. In this paper, we consider a novel class of matrix and tensor-valued features, which can be pre-trained using unlabeled samples. We present efficient algorithms for extracting discriminative information, given these pre-trained features and labeled samples for any related task. Our class of features are based on higher-order score functions, which capture local variations in the probability density function of the input. We establish a theoretical framework to characterize the nature of discriminative information that can be extracted from score-function features, when used in conjunction with labeled samples. We employ efficient spectral decomposition algorithms (on matrices and tensors) for extracting discriminative components. The advantage of employing tensor-valued features is that we can extract richer discriminative information in the form of an overcomplete representations. Thus, we present a novel framework for employing generative models of the input for discriminative learning.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Kernelized Stein Discrepancy for Goodness-of-fit Tests (2016, neutral): “This suggests a straightforward goodness-of-ﬁt testing procedure: Denote by F n ˆ S u the CDF of n ˆ S u under the null p = q , and set γ 1 − α the 1 − α quantile of F n ˆ S u , i.e., γ 1 − α = inf { s : F n ˆ S u ( s ) ≥ 1 − α } , then we reject the null with signiﬁcant level α if n ˆ S u ≥ γ 1 −…”
Generalization Bounds for Neural Networks through Tensor Factorization (2015, methodology): “Therefore, we can use any of these methods for estimating S 1 ( x ) and use the recursive form (Janzamin et al., 2014) to estimate higher order score functions.”
TensorLy: Tensor Learning in Python (2016, background): “Examples of such applications include, speech, music, image, communications, biomedical, and social network signal processing and analysis, as well as clustering, dimensionality reduction, subspace, dictionary and features learning e.g., [23, 24, 30, 22, 20, 25, 14, 2].”
Learning deep kernels for exponential family densities (2018, methodology): “, 2015), and learning discriminative features (Janzamin et al., 2014).”
Learning Two Layer Rectified Neural Networks in Polynomial Time (2018, methodology): “The first is that standard analysis of these methods utilizes a generalized version of Stein’s Lemma to compute the expected value of the tensor, which relies on the smoothness of the activation function.”
Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods (2017, methodology): “Proof of Lemma 6: It is shown by Janzamin et al. (2014) that the score function yields differ- ential operator such that for label-function f(x) := E[y|x], we have E[y · S3(x)] = E[∇(3)x f(x)].”