Are Discrete Units Necessary for Spoken Language Modeling?
By Tu Nguyen, Benoît Sagot, Emmanuel Dupoux. 2022
35 citations recorded by the source corpus.
Recent work in spoken language modeling shows the possibility of learning a language unsupervisedly from raw audio without any text labels. The approach relies first on transforming the audio into a sequence of discrete units (or pseudo-text) and then training a language model directly on such pseudo-text. Is such a discrete bottleneck necessary, potentially introducing irreversible errors in the encoding of the speech signal, or could we learn a language model without discrete units at all? In this work, we study the role of discrete versus continuous representations in spoken language modeling. We show that discretization is indeed essential for good results in spoken language modeling. We show that discretization removes linguistically irrelevant information from the continuous features, helping to improve language modeling performances. On the basis of this study, we train a language model on the discrete units of the HuBERT features, reaching new state-of-the-art results in the lexical, syntactic and semantic metrics of the Zero Resource Speech Challenge 2021 (Track 1 - Speech Only).
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Neil Zeghidour
- Marco Tagliasacchi
- Zalán Borsos
- Matthew Sharifi
- Damien Vincent
- Tom Ko
- Mingxuan Wang
- Evgeny Kharitonov
In Computer Vision and Pattern Recognition
- Neil Zeghidour
- Mingxuan Wang
- Olivier Pietquin
- Zalán Borsos
- Evgeny Kharitonov
- Marco Tagliasacchi
- Qiao Tian
- Matthew Sharifi
In Natural Language Processing
- Neil Zeghidour
- Mingxuan Wang
- Zalán Borsos
- Evgeny Kharitonov
- Olivier Pietquin
- Marco Tagliasacchi
- Qiao Tian
- Matthew Sharifi
In Speech & Audio
- Neil Zeghidour
- Zalán Borsos
- Marco Tagliasacchi
- Matthew Sharifi
- Damien Vincent
- Tom Ko
- Evgeny Kharitonov
- Chutong Meng
Citations in the corpus
- RepCodec: A Speech Representation Codec for Speech Tokenization (2023, background): “By converting speech into discrete tokens, language models can predict the future semantic content and generate realistic speech with long-term consistency (Nguyen et al., 2022).”
- PolyVoice: Language Models for Speech to Speech Translation (2023, background): “However, recent advances in the GPT family (Brown et al., 2020; Ouyang et al., 2022) have demonstrated the powerful capability of language modeling by the decoder-only architecture.”
- AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “We also add an improved CPC-BERT model by Nguyen et al. [59].”