A segmental framework for fully-unsupervised large-vocabulary speech recognition

By Herman Kamper, Aren Jansen, Sharon Goldwater. 2016

106 citations recorded by the source corpus.

Zero-resource speech technology is a growing research area that aims to develop methods for speech processing in the absence of transcriptions, lexicons, or language modelling text. Early term discovery systems focused on identifying isolated recurring patterns in a corpus, while more recent full-coverage systems attempt to completely segment and cluster the audio into word-like units---effectively performing unsupervised speech recognition. This article presents the first attempt we are aware of to apply such a system to large-vocabulary multi-speaker data. Our system uses a Bayesian modelling framework with segmental word representations: each word segment is represented as a fixed-dimensional acoustic embedding obtained by mapping the sequence of feature frames to a single embedding vector. We compare our system on English and Xitsonga datasets to state-of-the-art baselines, using a variety of measures including word error rate (obtained by mapping the unsupervised output to ground truth transcriptions). Very high word error rates are reported---in the order of 70--80% for speaker-dependent and 80--95% for speaker-independent systems---highlighting the difficulty of this task. Nevertheless, in terms of cluster quality and word segmentation metrics, we show that by imposing a consistent top-down segmentation while also using bottom-up knowledge from detected syllable boundaries, both single-speaker and multi-speaker versions of our system outperform a purely bottom-up single-speaker syllable-based approach. We also show that the discovered clusters can be made less speaker- and gender-specific by using an unsupervised autoencoder-like feature extractor to learn better frame-level features (prior to embedding). Our system's discovered clusters are still less pure than those of unsupervised term discovery systems, but provide far greater coverage.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Unsupervised Speech Recognition (2021, methodology): “There has been a lot of prior work in unsupervised speech segmentation (Kamper et al., 2017a,b; Rasanen et al., 2015; Kreuk et al., 2020) but here we simply use a method based on clustering the wav2vec 2.0 speech representations c1, . . . , cT .”
Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction (2020, background): “, [14, 15, 16, 17]), where the focus is to learn phonetic or word-like units, or representations that can distinguish among such units.”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “A different line of work employs probabilistic graphical models to jointly segment and cluster the speech signal (Varadarajan et al., 2008; Zhang & Glass, 2009; Gish et al., 2009; Lee & Glass, 2012; Siu et al., 2014; Lee et al., 2015; Ondel et al., 2016; Kamper et al., 2016; 2017a).”
wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “There has been work on unsupervised learning for speech but the resulting representations have not been applied to improve supervised speech recognition (Synnaeve & Dupoux, 2016b; Kamper et al., 2017; Chung et al., 2018; Chen et al., 2018; Chorowski et al., 2019).”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, background): “segments into a common vector space where they could be clustered into word types [75].”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, background): “It would be interesting to explore less supervised seg-mentations to learn word boundaries [47, 48].”
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input (2018, methodology): “…utilized an audio networkwhichwas trained from a random initialization, but used a vision model which underwent supervised pre-training e.g. on ImageNet (Harwath et al. 2016; Harwath and Glass 2017; Gelderloos and Chrupala 2016; Chrupala et al. 2017; Alishahi et al. 2017; Kamper et al. 2017).”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, methodology): “The second one, called embedded segmental K-means model (ES-KMeans) [36], is an approximation to BES-GMM that uses hard clustering and segmentation, rather than full Bayesian inference.”
An embedded segmental K-means model for unsupervised segmentation and clustering of speech (2017, methodology): “In previous work [20, 23], we proposed a very similar model, but instead of K-means, we used a Bayesian GMM as wholeword clustering component (top of Fig.”
Exploring TTS without T Using Biologically/Psychologically Motivated Neural Network Modules (ZeroSpeech 2020) (2020, background): “A popular approach to such clustering problems in computational linguistics and cognitive science is Bayesian clustering based on the Dirichlet distribution/process [6, 7, 8, 9, 10, 11, 12, 13, 14].”
Measuring the perceptual availability of phonological features during language acquisition using unsupervised binary stochastic autoencoders (2019, methodology): “Following preceding work showing improved unsupervised clustering when segments are given fixed-dimensional acoustic representations, thus abstracting away from the variable temporal dilation in natural speech (Kamper et al., 2017a,b), we resample all reconstruction targets to a length of 25 frames.”
Analysis of Audio-Visual Features for Unsupervised Speech Recognition (2017, methodology): “For all audio-visual features, we reduced the covariance scale of the discovered clusters to 1 e − 5 (we used 1 e − 3 for the MFCC features, as in [4]).”
Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech (2018, methodology): “In [3], embedding models for images and audio captions were shown to be capable of performing semantic retrieval tasks, and more recent works have studied word and object discovery [4] and keyword spotting [8].”
Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings (2018, background): “This is why unsupervised approaches of speech processing, including automatically discovering acoustic tokens and learning representations and linguistic structures from unlabeled audio corpora is attractive, desirable and critical [3].”
Audio Word2vec: Sequence-to-Sequence Autoencoding for Unsupervised Learning of Audio Segmentation and Representation (2019, background): “1 Probabilistic Bayesian models are developed to construct a model which learns segmentation and representation (or clustering) jointly [21], [25], [64].”
Learning Word Embeddings: Unsupervised Methods for Fixed-size Representations of Variable-length Speech Segments (2018, methodology): “In [11], this approach is replaced by a simpler procedure, where a fixedsize embedding is extracted from a sequence of frames by picking 10 equally-spaced frames from the sequence.”
Pre-training on high-resource speech recognition improves low-resource speech-to-text translation (2018, background): “On the speech side, it might be even more effective to use multilingual training; or to replacing the MFCC input features with pre-trained multilingual features, or features that are targeted to low-resource multispeaker settings [47,48].”
Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection (2018, methodology): “It seems the performance of the spoken word segmentation has to be above some minimum level, otherwise the audio Word2Vec couldn’t be reasonably trained, or spoken word segmentation boundaries had the major impact on the STD performance.”
Speech Recognition (2018, unknown): “There has been work on unsupervised learning for speech but the resulting representations have not been applied to improve supervised speech recognition (Synnaeve & Dupoux, 2016b; Kamper et al., 2017; Chung et al., 2018; Chen et al., 2018; Chorowski et al., 2019).”