Uncertainty-Aware Audiovisual Activity Recognition Using Deep Bayesian Variational Inference

By Mahesh Subedar, Ranganath Krishnan, Paulo López-Meyer, Omesh Tickoo, Jonathan Huang. 2018

84 citations recorded by the source corpus.

Deep neural networks (DNNs) provide state-of-the-art results for a multitude of applications, but the approaches using DNNs for multimodal audiovisual applications do not consider predictive uncertainty associated with individual modalities. Bayesian deep learning methods provide principled confidence and quantify predictive uncertainty. Our contribution in this work is to propose an uncertainty aware multimodal Bayesian fusion framework for activity recognition. We demonstrate a novel approach that combines deterministic and variational layers to scale Bayesian DNNs to deeper architectures. Our experiments using in- and out-of-distribution samples selected from a subset of Moments-in-Time (MiT) dataset show a more reliable confidence measure as compared to the non-Bayesian baseline and the Monte Carlo dropout (MC dropout) approximate Bayesian inference. We also demonstrate the uncertainty estimates obtained from the proposed framework can identify out-of-distribution data on the UCF101 and MiT datasets. In the multimodal setting, the proposed framework improved precision-recall AUC by 10.2% on the subset of MiT dataset as compared to non-Bayesian baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, background): “[33] proposed uncertainty aware variational layers for activity recognition.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “, multi-modal Bayesian fusion [90], multi-level fusion [471], image fusion [472], data fusion [473], etc.”
Deep Learning for Geophysics: Current and Future Trends (2021, methodology): “, 2019), variational inference (Subedar et al., 2019), etc.”
What Makes Multimodal Learning Better than Single (Provably) (2021, background): “It has been discovered that the use of multi-modal data in practice will degrade the performance of the model in some cases[43, 19, 38, 15].”
Stochastic Classifiers for Unsupervised Domain Adaptation (2020, background): “For example, [50] proposed an uncertainty aware multi-modal BNNs for activity recognition and [59] modelled the feature uncertainty using distributions in person re-identification.”
Trusted Multi-View Classification With Dynamic Evidential Fusion (2022, background): “Recently, confidence-based fusion has attracted attention [63], [64].”