Early versus late fusion in semantic video analysis

By Cees G. M. Snoek, Marcel Worring, Arnold M. W. Smeulders. ACM Multimedia, 2005

987 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/3037897d2fd1cc72dfc5a5b79cf9c0e8cdae083e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Parallel field alignment for cross media retrieval (2013, background): “Aiming at this point, many previous works [27] have been proposed to simultaneously utilize multiple types of information such as the original multimedia contents, surrounding texts (or labels), and links to improve the multimedia retrieval performance.”
Extensive complementarity between gene function prediction methods (2016, background): “One approach to this is 'early fusion', which would imply joining the five sets of features together before having constructed classification models (Snoek et al., 2005; Dong et al., 2014).”
Learning latent spatio-temporal compositional model for human action recognition (2013, background): “rich amount of multimedia documents surrounding us. To interact with these videos, there have been increasing demands of understanding human activities in these videos. Although many research studies [45, 41, 31, 46, 22, 5, 29, 37, 32, 8] have been carried out to understand and retrieve large scale ”
Color attributes for object detection (2012, neutral): “On this dataset, our approach yields a significant gain of 14% in mean AP over conventional state-of-the-art methods.”
IntentNet: Learning to Predict Intention from Raw Sensor Data (2018, background): “Work across different domains [30, 31, 32] has shown that late fusion delivers stronger performance than early fusion.”
A discriminative CNN video representation for event detection (2014, methodology): “We investigate average late fusion [39] to fuse the prediction results from different layers with PQ compression, i.”
Concept-Based Video Retrieval (2009, methodology): “The classical method to obtain separate classifiers is to learn the same concept detector on several unimodal [109, 240, 257, 267, 286] or multimodal features [135, 228, 272].”
Click Prediction for Web Image Reranking Using Multimodal Sparse Coding (2014, methodology): “:3) i , . . . , x ( t ) i , but this representation may reduce the performance of algorithms [20], especially when the features are independent or heterogeneous.”
A Formal Study of Shot Boundary Detection (2007, background): “compared them in the semantic video analysis [59].”
Deep Multimodal Fusion by Channel Exchanging (2020, background): “Another categorization of multimodal fusion can be specified as early, middle, and late fusion, depending on when to fuse, which have been discussed in earlier works [2, 7, 17, 41] and also in the deep learning literature [4, 26, 27, 45].”
Does My Multimodal Model Learn Cross-modal Interactions? It’s Harder to Tell than You Might Think! (2020, background): “3The multimedia commonly makes a related distinction between early fusion (joint multimodal processing) and late fusion (ensembles) (Snoek et al., 2005). is multimodally additive if it decomposes as f(t, v) = fT (t) + fV (v) (1) Additive multimodal models are simply ensembles of unimodal…”
Risk Prediction with Electronic Health Records: A Deep Learning Approach (2016, methodology): “Those methods typically use separate stacked video frames as input to the network, they try to combine those stacked videos by different fusion mechanisms on different layers of the CNN architecture [32, 33].”
Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals (2014, background): “In this class of fusion, features are integrated in the beginning (Cees et al., 2005).”
Extending Long Short-Term Memory for Multi-View Structured Learning (2016, background): “Two ways of fusing multi-view data are early and late fusion techniques [19].”
A deep learning system for differential diagnosis of skin diseases (2019, positive): “These two components were joined at the top using a fully connected layer (that is, late fusion 51 ).”