Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision

By Aren Jansen, D. Ellis, Shawn Hershey, R. Channing Moore, Manoj Plakal, Ashok Popat, Rif A. Saurous. 2019

32 citations recorded by the source corpus.

Humans do not acquire perceptual abilities in the way we train machines. While machine learning algorithms typically operate on large collections of randomly-chosen, explicitly-labeled examples, human acquisition relies more heavily on multimodal unsupervised learning (as infants) and active learning (as children). With this motivation, we present a learning framework for sound representation and recognition that combines (i) a self-supervised objective based on a general notion of unimodal and cross-modal coincidence, (ii) a clustering objective that reflects our need to impose categorical structure on our experiences, and (iii) a cluster-based active learning procedure that solicits targeted weak supervision to consolidate categories into relevant semantic classes. By training a combined sound embedding/clustering/classification network according to these criteria, we achieve a new state-of-the-art unsupervised audio representation and demonstrate up to a 20-fold reduction in the number of labels required to reach a desired classification performance.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, neutral): “[51] I3D (12.1M) HT 15 VT 83.4 89.1 54.8 59.2 / / MIL-NCE [51] S3D-G (9.1M) HT 15 VT 82.7 91.3 53.1 61.0 / / AVTS [43] MC3 (11.7M) AS 1 VA 89.0 61.6 80.6 AVTS [43] MC3 (11.7M) SNet 1 VA 82.3 AA+AV CC [34] RN-50 (23.5M) AS 1 VA 28.5 CVRL [70] R3D50 (33.3M) K600 0.1 V 64.1 XDC [6] R(2+1)D-18 (33.3M) A”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, methodology): “(5) This penalty is inspired by a similar loss used in image VQGAN model (Chang et al., 2022), which is also found in entropy-based clustering (Jansen et al., 2020). ř In LFQ, given the independence among dimensions, we rewrite H p q p z qq “ H p q p z i qq .”
Multimodal Self-Supervised Learning of General Audio Representations (2021, unknown): “It has later been extended to multimodal setting with an additional clustering-based loss, and an improvement on the AudioSet tagging task has been reported [19].”
Attention Bottlenecks for Multimodal Fusion (2021, background): “In the unsupervised setting, audiovisual learning is commonly used to learn good unimodal representations, with a popular pretraining task being to synchronise signals from different modalities via a contrastive loss [4, 5, 7, 46, 30, 2, 3], however each modality is usually encoded separately under ”
Multi-Format Contrastive Learning of Audio Representations (2021, methodology): “We evaluate the representations in the downstream task of training shallow fully connected audio classifiers following the same setup as in [20, 26], where a 1-hidden-layer MLP with 512 units is used and the parameters in the pretrained network are fixed.”