Learning with Side Information through Modality Hallucination

By Judy Hoffman, Saurabh Gupta, Trevor Darrell. 2016

229 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “Hoffman et al. (2016) proposed a modality hallucination architecture based on cross-modality distillation to improve detection performance.”
Learning and Refining of Privileged Information-Based RNNs for Action Recognition from Depth Sequences (2017, background): “edge, also referred to as privileged information (PI) [41], hidden information [50] or side information [54, 19], is only available during training but not available during testing.”
Multi-view 3D Object Detection Network for Autonomous Driving (2016, methodology): “To combine information from different features, prior work usually use early fusion [1] or late fusion [19, 11].”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “osed by a set of crawled Web pages containing both text and images. (Image courtesy B. Tan [222]). web pages or photos with tags or comments) [110, 210, 256] or images acquired with depth information [115]. We can also have multi-view situations even when the views have the same data modalities (tex”
End-to-End Learning of Driving Models from Large-Scale Video Datasets (2016, background): “We propose to exploit privileged learning [24, 22, 8] to learn a driving policy that exploits both task loss and available side losses.”
Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networks (2017, unknown): “Therefore, improving the method to be robust to impure data and artifacts could be helpful, e.g. by using hallucination networks [45] to infer the missing modality as proposed in [46].”
Improving One-Shot Learning through Fusing Side Information (2017, background): “Hoffman et al. (2016) presented a modality hallucination architecture for RGB image detection ob-jective by incorporating depth of the images as side information.”
Graph Distillation for Action Detection with Privileged Modalities (2017, methodology): “This approach was adopted by [19] and [15] in cross-modality knowledge transfer.”
MARS: Motion-Augmented RGB Stream for Action Recognition (2019, unknown): “[15] hallucinate depth information for an object detector by combining different losses between mid-level features with standard object detection losses.”
Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations (2018, background): “[39] and Li & Hoiem [40] only consider a single task learning (object detection and image classification, respectively).”
Teaching Compositionality to CNNs (2017, background): “Examples include learning representations from video sequences [2, 10, 17], encouraging the utilization of depth information [14], and using physical interaction with the eninput images”
PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition (2018, background): “Previous works often use early fusion [2] or late fusion [13, 27] to handle it.”
Cross-Modality Distillation: A Case for Conditional Generative Adversarial Networks (2018, methodology): “This kind of distillation was then extended to training networks to deal with missing modalities in [2], where L(2) loss (hallucination loss) is used to train a hallucination network.”
Modality Distillation with Multiple Stream Networks for Action Recognition (2018, methodology): “We also introduce and learn a hallucination network [2], using a new learning paradigm, loss function and design mechanism.”
Label Efficient Learning of Transferable Representations acrosss Domains and Tasks (2017, result): “Weight sharing of CNN parameters [58, 22, 21, 3] and minimizing the distribution discrepancy of network activations [51, 65, 30] have also shown convincing results.”
Domain Generalization: A Survey (2021, background): “, generic object detection [226] or human detection [227].”
Cross-View Policy Learning for Street Navigation (2019, methodology): “The hallucination network in [18] was trained on an existing modality to regress the missing modality using L2 loss, and leveraged multiple such losses for multiple tasks.”