Towards Learning a Universal Non-Semantic Representation of Speech
By Joel Shor, Aren Jansen, Ronnie Maor, Oran Lang, Omry Tuval, Felix de Chaumont Quitry, Marco Tagliasacchi, Ira Shavitt, Dotan Emanuel, Yinnon A. Haviv. 2020
174 citations recorded by the source corpus.
The ultimate goal of transfer learning is to reduce labeled data requirements by exploiting a pre-existing embedding model trained for different datasets or tasks. The visual and language communities have established benchmarks to compare embeddings, but the speech community has yet to do so. This paper proposes a benchmark for comparing speech representations on non-semantic tasks, and proposes a representation based on an unsupervised triplet-loss objective. The proposed representation outperforms other representations on the benchmark, and even exceeds state-of-the-art performance on a number of transfer learning tasks. The embedding is trained on a publicly available dataset, and it is tested on a variety of low-resource downstream tasks, including personalization tasks and medical domain. The benchmark, models, and evaluation code are publicly released.
Subfields: Computer Vision and Pattern Recognition, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jesse Engel
- Trevor Darrell
- Dacheng Tao
- Li Fei-Fei
- Cordelia Schmid
- Neil Zeghidour
- Abhinav Kumar Gupta
In Signal Processing
- Hung-yi Lee
- Neil Zeghidour
- Shinji Watanabe
- James R. Glass
- Marco Tagliasacchi
- Abdel-rahman Mohamed
- Shang-Wen Li
- Xuankai Chang
Citations in the corpus
- SUPERB: Speech processing Universal PERformance Benchmark (2021, methodology): “There are existing benchmarks proposed to evaluate representations extracted from SSL pretrained models [21, 22].”
- MusicLM: Generating Music From Text (2023, methodology): “We report the FAD based on two audio embedding models, both of which are publicly available: (1) Trill2 (Shor et al., 2020), which is trained on speech data, and (2) VGGish3, (Hershey et al.”
- FRILL: A Non-Semantic Speech Embedding for Mobile Devices (2020, unknown): “Using each spectrogram, we compute the layer19 output of the TRILL model, as was done in [4].”
- Multimodal Self-Supervised Learning of General Audio Representations (2021, unknown): “In [18, 22], a triplet loss is employed to minimize the distance between the anchor and positives.”
- Contrastive Learning of General-Purpose Audio Representations (2020, methodology): “We also compare with the recent self-supervised learning framework TRILL [13] on three speech-related tasks, benchmarking against TRILL-19 (the best self-supervised system of [13]).”
- Comparing Supervised Models And Learned Speech Representations For Classifying Intelligibility Of Disordered Speech On Selected Phrases (2021, methodology): “We then train a Logistic Regression model on the embeddings to predict intelligibility, using multinomial loss (identical to [1]) to get a distribution over classes.”
- Pengi: An Audio Language Model for Audio Tasks (2023, methodology): “In the domain of audio processing, Transfer Learning has facilitated the rise of Self-Supervised Learning and Zero-Shot Learning techniques [52, 53, 45, 24, 27, 26, 3, 17, 28, 25, 59, 60, 43, 13, 16, 18, 65, 10].”