Online Model Distillation for Efficient Video Inference

By Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, Kayvon Fatahalian. 2018

132 citations recorded by the source corpus.

High-quality computer vision models typically address the problem of understanding the general distribution of real-world images. However, most cameras observe only a very small fraction of this distribution. This offers the possibility of achieving more efficient inference by specializing compact, low-cost models to the specific distribution of frames observed by a single camera. In this paper, we employ the technique of model distillation (supervising a low-cost student model using the output of a high-cost teacher) to specialize accurate, low-cost semantic segmentation models to a target video stream. Rather than learn a specialized student model on offline data from the video stream, we train the student in an online fashion on the live video, intermittently running the teacher to provide a target for learning. Online model distillation yields semantic segmentation models that closely approximate their Mask R-CNN teacher with 7~to~17$\times$ lower inference runtime cost (11~to~26$\times$ in FLOPs), even when the target video's distribution is non-stationary. Our method requires no offline pretraining on the target video stream, achieves higher accuracy and lower cost than solutions based on flow or video object segmentation, and can exhibit better temporal stability than the original teacher. We also provide a new video dataset for evaluating the efficiency of inference over long running video streams.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “2020b), image/video segmentation (He et al. 2019; Mullapudi et al. 2019; Dou et al. 2020; Liu et al. 2019h; Siam et al. 2019; Hou et al. 2020; Bergmann et al. 2020), action recognition (Luo et al.”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “[35] Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian.”
Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024, unknown): “In computer vision, the idea of learning at test time has been applied for decades to applications such as face detection [41], object detection [56], image super-resolution [68], and 3D reconstruction [53].”
Test-Time Training for Out-of-Distribution Generalization (2019, background): “2018), which makes video segmentation more efﬁcient by using a student model that learns online from a teacher model.”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “Distillation serves to compress models [29, 49, 8] or incorporate privileged information from alternate tasks [43].”
Space-Time Correspondence as a Contrastive Random Walk (2020, background): “A key beneﬁt of self-supervised learning is that, because there is no reliance on labeled data, training need not be limited to the training phase, but can continue during deployment [3, 56, 75].”
Side-Tuning: Network Adaptation via Additive Side Networks (2019, unknown): “[17]).”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “tion, many OTTA methods [490], [492], [511], [515], [516], [517] adopt pseudo labels generated at test time for model updates.”
Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks (2019, unknown): “in [20]).”
Test-Time Training with Self-Supervision for Generalization under Distribution Shifts (2019, methodology): “Note that some of the previously covered papers (Hoffman et al., 2014; Jain & Learned-Miller, 2011; Mullapudi et al., 2018) use the term “online learning” outside of the learning theory setting, so the term can be overloaded.”