Variable-rate discrete representation learning

By Sander Dieleman, Charlie Nash, Jesse Engel, Karen Simonyan. 2021

36 citations recorded by the source corpus.

Semantically meaningful information content in perceptual signals is usually unevenly distributed. In speech signals for example, there are often many silences, and the speed of pronunciation can vary considerably. In this work, we propose slow autoencoders (SlowAEs) for unsupervised learning of high-level variable-rate discrete representations of sequences, and apply them to speech. We show that the resulting event-based representations automatically grow or shrink depending on the density of salient information in the input signals, while still allowing for faithful signal reconstruction. We develop run-length Transformers (RLTs) for event-based representation modelling and use them to construct language models in the speech domain, which are able to generate grammatical and semantically coherent utterances and continuations.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Generative Spoken Dialogue Language Modeling (2022, background): “The discrete representation can in turn be fed to a standard autoregressive language model, which can then be sampled to generate new speech sequences (Lakhotia et al., 2021; Dieleman et al., 2021).”
ShapeFormer: Transformer-based Shape Completion via Sparse Representation (2022, methodology): “However, for representations that contain enormous statistical redundancy, as in the case of current local methods, such combination [60] excessively allocates model capacity towards perceptually irrelevant details [22, 26].”
Variable Bitrate Discrete Neural Representations via Causal Self-Attention (2021, methodology): “SlowAE [6] produces event-based representations by outputting features at each time-step t , then quantizing the features at each time-step and compressing via run-length encoding across the time dimension.”
Continuous diffusion for categorical data (2022, methodology): “…with L2 regularisation, margin penalties (Dieleman et al., 2021; Donahue et al., 2019), as well as automatic adaptation of the L2 penalty weight to target unit variance for the embedding vector components (Dieleman et al., 2021; Rezende and Viola, 2018), but found that normalisation works best.”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “In practice, this is often achieved implicitly, by learning segment boundaries or by taking repeated quantized values to belong to the same segment (Kamper and van Niekerk 2021; Chorowski et al. 2019a; Michel et al. 2017; Kreuk, Keshet, and Adi 2020; Wang, Chung, and Lee 2017; Dieleman et al. 2021).”