Discrete Autoencoders for Sequence Models

By Lukasz Kaiser, Samy Bengio. 2018

54 citations recorded by the source corpus.

Recurrent models for sequences have been recently successful at many tasks, especially for language modeling and machine translation. Nevertheless, it remains challenging to extract good representations from these models. For instance, even though language has a clear hierarchical structure going from characters through words to sentences, it is not apparent in current language models. We propose to improve the representation in sequence models by augmenting current approaches with an autoencoder that is forced to compress the sequence through an intermediate discrete latent space. In order to propagate gradients though this discrete representation we introduce an improved semantic hashing technique. We show that this technique performs well on a newly proposed quantitative efficiency measure. We also analyze latent codes produced by the model showing how they correspond to words and phrases. Finally, we present an application of the autoencoder-augmented model to generating diverse translations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Model Architectures

Citations in the corpus

Sparse is Enough in Scaling Transformers (2021, neutral): “Sparse attention-based approaches have made the attention layer more efﬁcient, especially for long sequences, by incorporating additional combinatorial mechanisms, as in [40], or selecting a subset of tokens this layer attends to [31, 5, 19, 37, 15, 4] or other approaches [12].”
Dynamic Neural Networks: A Survey (2021, methodology): “An alternative technique is Improved SemHash [261] adopted in [88] and [90] to train their hard gating modules.”
Model-Based Reinforcement Learning for Atari (2019, methodology): “To keep the model differentiable, the backpropagation bypasses the discretization following Kaiser & Bengio (2018).”
Fast Decoding in Sequence Models using Discrete Latent Variables (2018, methodology): “Three techniques recently have shown how to successfuly use discrete variables in deep models: the Gumbel-Softmax (Jang et al., 2016; Maddison et al., 2016), VQ-VAE (van den Oord et al., 2017) and improved semantic hashing (Kaiser & Bengio, 2018).”
Encoding Musical Style with Transformer Autoencoders (2019, methodology): “Most similar to our work is that of (Kaiser & Bengio, 2018), which uses a Transformer decoder and a discrete autoencoding function to map an input sequence into a discretized, compressed representation.”
You Look Twice: GaterNet for Dynamic Filter Selection in CNNs (2018, methodology): “In this paper, we adopt a method called Improved SemHash [17, 18].”
Dynamic Channel and Layer Gating in Convolutional Neural Networks (2020, methodology): “employed Improved SemHash trick[12] to make discrete gate functions differentiable during the backward pass.”
Theory and Experiments on Vector Quantized Autoencoders (2018, methodology): “…reconstructed images from CIFAR-10 using VQ-VAE trained using EM with a code-book of size 2 [32] with no beam search, NAT denotes the Non-Autoregressive Transformer of [6], and LT + Semhash denotes the Latent Transformer from [31] using the improved semantic hashing discretization technique of[9].”
COD3S: Diverse Generation with Discrete Semantic Signatures (2020, methodology): “ing codes produced from improved semantic hashing (Kaiser and Bengio, 2018) with a TreeLSTMbased autoencoder.”
Self-Adaptive Network Pruning (2019, methodology): “We adopt a recently proposed discretization technique namely Improved Semantic Hashing Kaiser & Bengio (2018) to generate channel pruning strategy from saliency scores sl(xl−1).”