Learning Weakly Supervised Multimodal Phoneme Embeddings

By Rahma Chaabouni, Ewan Dunbar, Neil Zeghidour, Emmanuel Dupoux. 2017

12 citations recorded by the source corpus.

Recent works have explored deep architectures for learning multimodal speech representation (e.g. audio and images, articulation and audio) in a supervised way. Here we investigate the role of combining different speech modalities, i.e. audio and visual information representing the lips movements, in a weakly supervised way using Siamese networks and lexical same-different side information. In particular, we ask whether one modality can benefit from the other to provide a richer representation for phone recognition in a weakly supervised setting. We introduce mono-task and multi-task methods for merging speech and visual modalities for phone recognition. The mono-task learning consists in applying a Siamese network on the concatenation of the two modalities, while the multi-task learning receives several different combinations of modalities at train time. We show that multi-task learning enhances discriminability for visual and multimodal inputs while minimally impacting auditory inputs. Furthermore, we present a qualitative analysis of the obtained phone embeddings, and show that cross-modal visual input can improve the discriminability of phonological features which are visually discernable (rounding, open/close, labial place of articulation), resulting in representations that are closer to abstract linguistic features than those based on audio only.

Subfields: Experimental and Cognitive Psychology, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Experimental and Cognitive Psychology

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “, 2015) Hidden activations in feed-forward acoustic model Phonemes, phonetic features, gender Clustering, average activations by group/label (Chaabouni et al., 2017) Hidden activations in feed-forward audiovisual model Phonetic features Discrimination”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “A joint audio-visual model of speech and lip movements was developed in [18], where phoneme embeddings were shown to be closer to certain linguistic features than embeddings based on audio alone.”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “This may be computed directly, for instance by computing the correlation between long short-term memory (LSTM) cell activations and Mel-frequency cepstral coefficient (MFCC) acoustic features [345], or indirectly, by defining discrimination tasks based on activations [5, 57].”