Learning using privileged information: similarity control and knowledge transfer

By Vladimir Naumovich Vapnik, Rauf Izmailov. 2015

446 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Improved Recurrent Neural Networks for Session-based Recommendations (2016, methodology): “The learning using privileged information (LUPI) framework [28, 27] was proposed to utilize the additional feature representations that are only available during training but not during testing.”
Model compression via distillation and quantization (2018, background): “More generally, it can be seen as a special instance of learning with privileged information, e.g. Vapnik & Izmailov (2015); Xu et al. (2016), in which the student is provided additional information in the form of outputs from a larger, pre-trained model.”
Learning Credible Models (2017, methodology): “…w ∗ | θ | [ i ] is the i th largest parameter sorted by magnitude; and ⊙ is the training time, Sun et al . explore using features identified as relevant Vapnik et al . explore the theory of learning with privileged information In the linear setting, the method simplifies to l 2 regularization on”
Knowledge Distillation: A Survey (2020, background): “Distilled knowledge can be categorized from different perspectives, such as structured knowledge of the data (Liu et al. 2019g; Chen et al. 2021; Peng et al. 2019a; Tung and Mori 2019; Tian et al. 2020), privileged information about input features (Lopez-Paz et al. 2016; Vapnik and Izmailov 2015).”
Lifelong GAN: Continual Learning for Conditional Image Generation (2019, background): “The teacher classifier normally would have more privileged information [31] compared with the student classifier.”
Active Long Term Memory Networks (2016, background): “This framework received a lot of recent interest and has been extended to what is called Generalized Distillation [38] to incorporate some theoretical results of Vapnik’s privileged information algorithm [39].”
Deep Mutual Learning (2017, background): “Recently, distillation has been connected more systematically to information learning theory [15] and SVM+ [25] – an intelligent teacher provides privileged information to the student.”
Unifying distillation and privileged information (2015, background): “3.1 C OMPARISON TO PRIOR WORK Vapnik & Vashist (2009); Vapnik & Izmailov (2015) offer two strategies to learn using privileged information: similarity control and knowledge transfer.”
Data Distillation: Towards Omni-Supervised Learning (2017, background): “In [26] distillation is unified with privileged information [44].”
Algorithmic Fairness (2020, background): “They add constraints and regularization components to the privileged learning support vector machine (SVM) model proposed by [135].”
General Heterogeneous Transfer Distance Metric Learning via Knowledge Fragments Transfer (2017, methodology): “Inspired by the knowledge fragment transfer strategy [Vapnik and Izmailov, 2015], we develop a general heterogeneous TDML (HTDML) framework to overcome these defects.”
Transfer Hashing with Privileged Information (2016, background): “…of training data can be significantly reduced with privileged information for training a precise predictive model [Lapin et al., 2014; Vapnik and Vashist, 2009; Vapnik and Izmailov, 2015], we propose a new framework for learning to hash, namely Transfer Hashing with Privileged Information (THPI).”
Simple and Efficient Learning using Privileged Information (2016, methodology): “The SVM+ at-tracts much attention recently [Vapnik and Izmailov, 2015; Lopez-Paz et al. , 2015; Lapin et al. , 2014], and has been successfully applied to different applications [Niu et al. , 2016; Sharmanska et al. , 2013; Fouad et al. , 2012].”
MARS: Motion-Augmented RGB Stream for Action Recognition (2019, unknown): “To this end, we propose a solution based on the concept of learning under privileged information [39].”
Privileged Modality Distillation for Vessel Border Detection in Intracoronary Imaging (2019, methodology): “LUPI builds an extra knowledge source to provide useful information (i.e. privileged information) to conduce the learning of the target network in the learning phase, but applies the network without the privileged information in the test phase [28].”
Human-Guided Learning for Probabilistic Logic Models (2018, background): “A different, but related type of advice that we consider is the use of privileged information (Vapnik and Vashist, 2009; Vapnik and Izmailov, 2015), where some advice is available during training time that is not present during evaluation/deployment or testing times.”
Efficient Knowledge Distillation from an Ensemble of Teachers (2017, background): “In the knowledge distillation framework, instead of training models which had reduced computational requirements and improved latency performances directly on hard targets in a single step, training is now performed in two separate steps [4, 5, 6, 7, 8, 9, 10].”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, background): “David et al. unify the knowledge distillation with privileged information [53], [54].”