On Knowledge distillation from complex networks for response prediction
By Siddharth Arora, Mitesh M. Khapra, Harish Guruprasad Ramaswamy. 2019
16 citations recorded by the source corpus.
Recent advances in Question Answering have lead to the development of very complex models which compute rich representations for query and documents by capturing all pairwise interactions between query and document words. This makes these models expensive in space and time, and in practice one has to restrict the length of the documents that can be fed to these models. Such models have also been recently employed for the task of predicting dialog responses from available background documents (e.g., Holl-E dataset). However, here the documents are longer, thereby rendering these complex models infeasible except in select restricted settings. In order to overcome this, we use standard simple models which do not capture all pairwise interactions, but learn to emulate certain characteristics of a complex teacher network. Specifically, we first investigate the conicity of representations learned by a complex model and observe that it is significantly lower than that of simpler models. Based on this insight, we modify the simple architecture to mimic this characteristic. We go further by using knowledge distillation approaches, where the simple model acts as a student and learns to match the output from the complex teacher network. We experiment with the Holl-E dialog data set and show that by mimicking characteristics and matching outputs from a teacher, even a simple network can give improved performance.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Knowledge Distillation: A Survey (2020, background): “ranslation(NMT) (Hahn and Choi,2019; Kim and Rush, 2016;Zhou et al., 2019a; Tan et al.,2019; Gordon and Duh, 2019; Freitag et al., 2017; Wei et al., 2019), question answeringsystem(Wang et al.,2018c; Arora et al., 2019; Yang et al., 2020; Hu et al., 2018), document retrieval (Shakeri et al., 2019),t”
- Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “On the other aspect, [12], [28], [51], [57], [114], [118], [161], [174], [175], [182], [191], [201], [208], [210], [218], [231], [248], [258], [267] propose to add more teachers or other auxiliaries such as teaching assistant and small students to improve the robustness of ensemble distribution.”