Knowledge Adaptation: Teaching to Adapt

By Sebastian Ruder, Parsa Ghaffari, John Gerard Breslin. 2017

56 citations recorded by the source corpus.

Domain adaptation is crucial in many real-world applications where the distribution of the training data differs from the distribution of the test data. Previous Deep Learning-based approaches to domain adaptation need to be trained jointly on source and target domain data and are therefore unappealing in scenarios where models need to be adapted to a large number of domains or where a domain is evolving, e.g. spam detection where attackers continuously change their tactics. To fill this gap, we propose Knowledge Adaptation, an extension of Knowledge Distillation (Bucilua et al., 2006; Hinton et al., 2015) to the domain adaptation scenario. We show how a student model achieves state-of-the-art results on unsupervised domain adaptation from multiple sources on a standard sentiment analysis benchmark by taking into account the domain-specific expertise of multiple teachers and the similarities between their domains. When learning from a single teacher, using domain similarity to gauge trustworthiness is inadequate. To this end, we propose a simple metric that correlates well with the teacher's accuracy in the target domain. We demonstrate that incorporating high-confidence examples selected by this metric enables the student model to achieve state-of-the-art performance in the single-source scenario.

Subfields: Artificial Intelligence, Emergency Medicine

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Lin Wang
6. Kuk-Jin Yoon
7. W. Zhang
8. Yuang Liu

In Emergency Medicine

1. Christof Monz
2. Praveen Dakwale
3. Lin Wang
4. Kuk-Jin Yoon

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “More inter-estingly, due to the special characteristics of multi-teacher distillation, its extensions are used for domain adaptation via knowledge adaptation (Ruder et al. 2017), and to protect the privacy and security of data (Vongkulbhisal et al. 2019; Papernot et al. 2017).”
• Fine-Tuning for Neural Machine Translation with Limited Degradation across In- and Out-of-Domain Data (2017, methodology): “, 2015), “Knowledge adaptation” (Ruder et al., 2017) and “Learning without forgetting” (Li and Hoiem, 2016).”
• Adaptive multi-teacher multi-level knowledge distillation (2020, background): “[19] studies the domain adaption task by associating each source domain with a teacher for later distillation learning.”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “As a result, many new KD methods [9], [51], [56], [57], [80], [101], [114], [118], [131], [134], [139], [147], [155], [161], [174], [175], [182], [185], [186], [191], [195], [196], [201], [205], [208], [210], [214], [219], [231], [232], [234], [249], [252], [258], [263], [267], [272], [275] have bee”