Encoding of phonology in a recurrent neural model of grounded speech

By Afra Alishahi, Marie Barking, Grzegorz Chrupała. 2017

60 citations recorded by the source corpus.

We study the representation and encoding of phonemes in a recurrent neural network model of grounded speech. We use a model which processes images and their spoken descriptions, and projects the visual and auditory representations into the same semantic space. We perform a number of analyses on how information about individual phonemes is encoded in the MFCC features extracted from the speech signal, and the activations of the layers of the model. Via experiments with phoneme decoding and phoneme discrimination we show that phoneme representations are most salient in the lower layers of the model, where low-level signals are processed at a fine-grained level, although a large amount of phonological information is retain at the top recurrent layer. We further find out that the attention mechanism following the top recurrent layer significantly attenuates encoding of phonology and makes the utterance embeddings much more invariant to synonymy. Moreover, a hierarchical clustering of phoneme representations learned by the network shows an organizational structure of phonemes similar to those proposed in linguistics.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In Natural Language Processing

In Speech & Audio

In Signal Processing

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, background): “…abstractions, such as words and phones, which are learned by the models (Harwath & Glass, 2017; Harwath et al., 2018b; Drexler & Glass, 2017; Alishahi et al., 2017; Harwath et al., 2019; Harwath & Glass, 2019; Havard et al., 2019b), and the impact of jointly training with textual input…”
Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “, 2016b) and a visually grounded speech model (Alishahi et al., 2017).”
Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks (2017, background): “tain quantitative correlations between parts of the neural network and linguistic properties, in both speech (Wu and King, 2016; Alishahi et al., 2017; Belinkov and Glass, 2017; Wang et al., 2017) and language processing models (Köhn, 2015; Qian et al.”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “Other joint audio-visual models have also analyzed the learned representations in different ways [19, 20, 21].”
Analyzing analytical methods: The case of phonology in neural models of spoken language (2020, methodology): “…i.e. methods which use the activations from different layers of a deep learning architecture as input to a prediction model (e.g., Adi et al., 2017; Alishahi et al., 2017; Hupkes et al., 2018; Conneau et al., 2018); • Representational Similarity Analysis (RSA) borrowed from neuroscience…”
Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech (2018, background): “Other work has analyzed these models, and provided evidence that linguistic abstractions such as phones and words emerge in their internal representations [4, 5, 6, 7].”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, background): “A rich body of work has recently emerged investigating representation learning for speech using visual grounding objectives [92, 32, 37, 57, 39, 74, 15, 4, 85, 47, 58, 91, 54, 25], as well as how word-like and subword-like linguistic units can be made to emerge within these models [33, 36, 23, 4, 36”
Towards Visually Grounded Sub-word Speech Unit Discovery (2019, background): “While [21] and [22] examined the utility of the intermediate representations of visually-grounded speech models to perform tasks such as speaker, phoneme, and word discrimination, they did not investigate if and how discrete, sub-word units may be emerging within the models.”
Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech (2019, background): “Alishahi et al. (2017) studied if such models capture phonological information and showed that some layers do capture such information more accuratly than others.”
Transfer Learning from Audio-Visual Grounding to Speech Recognition (2019, background): “Another line of work has studied the use of the visual modality as a form of weak supervision using semantic information for acoustic modeling [20, 41, 42], followed up with analysis on representations learned from such models [27, 43, 25].”
Models of Visually Grounded Speech Signal Pay Attention to Nouns: A Bilingual Experiment on English and Japanese (2019, methodology): “[7] showed that a non trivial amount of phonological information is preserved in higher layers, and suggested that the attention layer focuses on semantic information.”
Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language (2018, methodology): “We follow the methodology of Alishahi et al. (2017b) and train an L2-penalized logistic regression model on the output of the S encoders for phonemes from 2,500 utterances and report clas-siﬁcation accuracies on data from 2,500 heldout utterances.”
Edinburgh Research Explorer Understanding Learning Dynamics Of Language Models with SVCCA (unknown): “The other end of the spectrum is focused on the structure of the representation space (e.g., the features and the property in question are linearly similar; Alishahi et al., 2017).”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, methodology): “Others have analyzed joint language-vision [123] or audio-vision models [5, 73, 142].”