Harp-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Coding

By Darius Petermann, Seungkwon Beack, Minje Kim. 2021

19 citations recorded by the source corpus.

We propose a novel autoencoder architecture that improves the architectural scalability of general-purpose neural audio coding models. An autoencoder-based codec employs quantization to turn its bottleneck layer activation into bitstrings, a process that hinders information flow between the encoder and decoder parts. To circumvent this issue, we employ additional skip connections between the corresponding pair of encoder-decoder layers. The assumption is that, in a mirrored autoencoder topology, a decoder layer reconstructs the intermediate feature representation of its corresponding encoder layer. Hence, any additional information directly propagated from the corresponding encoder layer helps the reconstruction. We implement this kind of skip connections in the form of additional autoencoders, each of which is a small codec that compresses the massive data transfer between the paired encoder-decoder layers. We empirically verify that the proposed hyper-autoencoded architecture improves perceptual audio quality compared to an ordinary autoencoder baseline.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Neil Zeghidour
2. Alexander Richard
3. Olivier Pietquin
4. Zalán Borsos
5. Yan Lu
6. Evgeny Kharitonov
7. Marco Tagliasacchi
8. Dejan Markovic

In Statistical and Nonlinear Physics

1. Zalán Borsos
2. Olivier Pietquin
3. Marco Tagliasacchi
4. David Grangier
5. Neil Zeghidour
6. Damien Vincent

In Signal Processing

1. Marco Tagliasacchi
2. Zalán Borsos
3. Neil Zeghidour
4. Matthew Sharifi
5. Evgeny Kharitonov
6. Damien Vincent
7. Xue Jiang
8. Xiulian Peng

Citations in the corpus

• Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec (2023, methodology): “Therefore, scalable codecs decomposing the fine-coarse structures of the encoded latent codes using residual VQ [25], multi-scale VQ [26], and crossscale VQ [27] or the output waveforms using cross-module residual learning [24] are introduced for tractable hierarchical codebooks.”
• AudioLM: A Language Modeling Approach to Audio Generation (2022, background): “Moreover, combining such highquality synthesis systems with differentiable quantization [22]– [24], allows training end-to-end neural codecs [16], [25]–[27] by compressing activations in a bottleneck layer.”
• Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, unknown): “In recent years, DNN aided compression algorithms have achieved significant results, often outperforming state-of-theart standardised codecs in a variety of source domains, from image [156]–[160], to video [161]–[169], speech [170], [171], and audio [172], [173] compression.”
• Cross-Scale Vector Quantization for Scalable Neural Speech Coding (2022, methodology): “Some researchers (a) Single-scale residual VQ [16] (b) Multi-scale VQ [19] (c) Our cross-scale VQ Figure 1: Scalable vector quantization. introduce residual vector quantization (RVQ [16]) into audio coding to achieve bitrate scalability [12, 14, 17, 18].”