Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN

By Jiyang Gao, Z. Li, Ramkant Nevatia. 2017

20 citations recorded by the source corpus.

Fine-grained image labels are desirable for many computer vision applications, such as visual search or mobile AI assistant. These applications rely on image classification models that can produce hundreds of thousands (e.g. 100K) of diversified fine-grained image labels on input images. However, training a network at this vocabulary scale is challenging, and suffers from intolerable large model size and slow training speed, which leads to unsatisfying classification performance. A straightforward solution would be training separate expert networks (specialists), with each specialist focusing on learning one specific vertical (e.g. cars, birds...). However, deploying dozens of expert networks in a practical system would significantly increase system complexity and inference latency, and consumes large amounts of computational resources. To address these challenges, we propose a Knowledge Concentration method, which effectively transfers the knowledge from dozens of specialists (multiple teacher networks) into one single model (one student network) to classify 100K object categories. There are three salient aspects in our method: (1) a multi-teacher single-student knowledge distillation framework; (2) a self-paced learning mechanism to allow the student to learn from different teachers at various paces; (3) structurally connected layers to expand the student network capacity with limited extra parameters. We validate our method on OpenImage and a newly collected dataset, Entity-Foto-Tree (EFT), with 100K categories, and show that the proposed model performs significantly better than the baseline generalist model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, background): “This label propagation process is sometimes referred to as label smearing [74] and its specifics are covered in Appendix D.”
Learning Metrics From Teachers: Compact Networks for Image Embedding (2019, background): “Network distillation can also be used to compress multiple teachers into a single student network [8].”
Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More (2019, methodology): “Similar to knowledge distillation, [6] proposes a multiteacher single-student knowledge concentration method to classify 100K object categories.”
The Benefit of Temporally-Strong Labels in Audio Event Classification (2021, background): “Preliminary investigation of this kind of “label smearing” (based on similar ideas from image classification [19, 20]) did not yield significant changes in metrics, but it warrants further attention.”
Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers (2019, result): “…of the proposed method against the following state-of-the-art approaches: HCP-2000 [Wei et al., 2014], HCP-VGG [Wei et al., 2016], RLSD+ft-RPN [Zhang et al., 2018], CNN+RNN [Wang et al., 2016], CNN+LSTM [Wang et al., 2017], PF-DLDL [Gao et al., 2017a] and the method of [Zhou et al., 2018].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “[59] utilize a similar methodology but focus on the same task (classification) with two teachers specialized in different classification problems.”
Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN (2020, background): “More recently, the work of [7] introduces a multi-teacher and single-student knowledge concentration approach.”
Safe Distillation Box (2021, background): “More recently, Gao et al. (2017) introduce a multi-teacher and single-student knowledge concentration approach.”