EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition

By Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, Dima Damen. 2019

403 citations recorded by the source corpus.

We focus on multi-modal fusion for egocentric action recognition, and propose a novel architecture for multi-modal temporal-binding, i.e. the combination of modalities within a range of temporal offsets. We train the architecture with three modalities -- RGB, Flow and Audio -- and combine them with mid-level fusion alongside sparse temporal sampling of fused representations. In contrast with previous works, modalities are fused before temporal aggregation, with shared modality fusion weights over time. Our proposed architecture is trained end-to-end, outperforming individual modalities as well as late-fusion of modalities. We demonstrate the importance of audio in egocentric vision, on per-class basis, for identifying actions as well as interacting objects. Our method achieves state of the art results on both the seen and unseen test sets of the largest egocentric dataset: EPIC-Kitchens, on all metrics using the public leaderboard.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “(Kazakos et al. 2019) learns multimodal embeddings (i.e., video + audio), and (Furnari and Farinella 2019) studies future action anticipation from egocentric videos.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “…video understanding Egocentric video offers a host of interesting challenges, such as human-object interactions [26,46,163], activity recognition [110,139,243], anticipation [4,75,86,144,205], video summarization [48, 129,131,147,148,232], detecting hands [16,134], parsing social interactions…”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “These use extracted features from TBN [49] trained on source.”
Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision (2020, background): “Therefore, audiovisual fusion can make full use of complementary information and become an extensive tendency in computer vision and speech recognition communities [1,2,32,26,10,19].”
PoliTO-IIT Submission to the EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition (2021, methodology): “In the other configurations, we adopted the mid-fusion strategy proposed in [8], to generate a common frameembedding among the modalities and used a Temporal Relation Module (TRM) [14] to aggregate features from different frames before feeding the final embeddings to the verb and noun classifiers.”
Egocentric Video-Language Pretraining (2022, background): “4% nDCG on video-text retrieval of EPIC-KITCHENS-100 [14] 1 , 32 .”
Parameter Efficient Multimodal Transformers for Video Representation Learning (2020, background): “…to access audio-visual embeddings only from the same time slices, which could be problematic when there is a temporally asynchronous relationship between the two modalities (e.g., a visual clip matches with sound captured a few times steps before) (Kazakos et al., 2019; Morgado et al., 2020).”
In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, background): “Our results are presented in Table 6 and compared to state-of-the-art methods [22], [61], [92], [116], [117].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Futhermore, multi-modal learning has been used for several applications such as lip reading [4, 20, 21], audio-visual source separation and localization [3, 7, 28, 29, 39, 77, 87, 91, 113, 114], speech recognition [2, 73, 84], efficient inference [30, 56], egocentric action recognition [51] and navi”
What Makes Training Multi-Modal Classification Networks Hard? (2019, methodology): “Comparing to published results, G-Blend uses less modalities (not using optical ﬂow as TBN Ensemble [29]), less backbones (Baidu-UTS [52] uses three 3D-CNNs plus two detection models), and a single model (TBN Ensemble [29] uses ensemble of ﬁve models).”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, methodology): “Multi-modal learning has been studied from multiple perspectives, such as two stream networks that fuse decisions from multiple modalities for classification [41, 7, 26, 27, 3], and cross-modal learning that takes one modality as input and make prediction on the other modality [29, 2, 62, 1, 15, 42]”
Balanced Multimodal Learning via On-the-fly Gradient Modulation (2022, background): “, action recognition [10, 23, 29], audiovisual speech recognition [16, 33], and visual question answering [2, 19].”
The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain (2020, background): “In recent years, progress has been made in many research areas related to human behavior understanding, such as action recognition [24, 55, 26, 68, 35, 40], object detection [28, 27, 52, 51] and human-object interaction detection [29, 32, 53, 44].”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, methodology): “Due to its large scale and diverse coverage of actions, Kinetics has widely been adopted as the standard dataset for pre-training both first- and third-person video models [27, 19, 35, 11].”
EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition 2021: Team M3EM Technical Report (2021, methodology): “We also use hand-object bounding boxes to crop the input images, and extract features using TBN without further fine-tuning as the cropped-RGB modality and croppedFlow modality.”