Finite Scalar Quantization: VQ-VAE Made Simple
By Fabian Mentzer, David C. Minnen, Eirikur Agustsson, Michael Tschannen. International Conference on Learning Representations, 2023
523 citations recorded by the source corpus.
We propose to replace vector quantization (VQ) in the latent representation of VQ-VAEs with a simple scheme termed finite scalar quantization (FSQ), where we project the VAE representation down to a few dimensions (typically less than 10). Each dimension is quantized to a small set of fixed values, leading to an (implicit) codebook given by the product of these sets. By appropriately choosing the number of dimensions and values each dimension can take, we obtain the same codebook size as in VQ. On top of such discrete representations, we can train the same models that have been trained on VQ-VAE representations. For example, autoregressive and masked transformer models for image generation, multimodal generation, and dense prediction computer vision tasks. Concretely, we employ FSQ with MaskGIT for image generation, and with UViM for depth estimation, colorization, and panoptic segmentation. Despite the much simpler design of FSQ, we obtain competitive performance in all these tasks. We emphasize that FSQ does not suffer from codebook collapse and does not need the complex machinery employed in VQ (commitment losses, codebook reseeding, code splitting, entropy penalties, etc.) to learn expressive discrete representations.
Subfields: Computer Vision and Pattern Recognition, Media Technology
View Paper: https://www.semanticscholar.org/paper/eb76781932d4cdc241f1d739f7a6e3d34a4eec87
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
Citations in the corpus
- Autoregressive Image Generation without Vector Quantization (2024, background): “But discrete tokenizers are difficult to train, which has recently drawn special focus [27, 54, 32].”
- An Image is Worth 32 Tokens for Reconstruction and Generation (2024, unknown): “Orthogonal to this, RQ-VAE [36] and MoVQ [76] study the effect of using multiple vector quantization steps per latent embedding, while MAGVIT-v2 [68] and FSQ [45] propose a lookup-free quantization.”
- CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models (2024, unknown): “As shown in Figure 1 (a), we insert the finite scalar quantization (FSQ) module [42] into the encoder of SenseVoice-Large ASR model [43].”