Regeneration Learning: A Learning Paradigm for Data Generation

By Xu Tan, Tao Qin, Jiang Bian, Tie-Yan Liu, Yoshua Bengio. 2023

20 citations recorded by the source corpus.

Machine learning methods for conditional data generation usually build a mapping from source conditional data X to target data Y. The target Y (e.g., text, speech, music, image, video) is usually high-dimensional and complex, and contains information that does not exist in source data, which hinders effective and efficient learning on the source-target mapping. In this paper, we present a learning paradigm called regeneration learning for data generation, which first generates Y' (an abstraction/representation of Y) from X and then generates Y from Y'. During training, Y' is obtained from Y through either handcrafted rules or self-supervised learning and is used to learn X-->Y' and Y'-->Y. Regeneration learning extends the concept of representation learning to data generation tasks, and can be regarded as a counterpart of traditional representation learning, since 1) regeneration learning handles the abstraction (Y') of the target data Y for data generation while traditional representation learning handles the abstraction (X') of source data X for data understanding; 2) both the processes of Y'-->Y in regeneration learning and X-->X' in representation learning can be learned in a self-supervised way (e.g., pre-training); 3) both the mappings from X to Y' in regeneration learning and from X' to Y in representation learning are simpler than the direct mapping from X to Y. We show that regeneration learning can be a widely-used paradigm for data generation (e.g., text generation, speech recognition, speech synthesis, music composition, image generation, and video generation) and can provide valuable insights into developing data generation methods.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Lei He
6. Frank K. Soong
7. Yanqing Liu
8. Yi Ren

Citations in the corpus

• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “Since speech waveform is complex and high-dimensional, following the paradigm of regeneration learning [33], we first convert speech waveform into latent vectors using the audio codec encoder and reconstruct speech waveform from the latent vectors using the audio codec decoder.”
• NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality (2022, methodology): “…[37], [38], [39] that uses VQ-VAE [20], [40], [41] to compress high-dimensional image into low-dimensional representations to ease the generation [42], we leverage VAE [22] to compress high-dimensional speech x into frame-level representations z (i.e., z is sampled fromposteriordistribution q (…”
• AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, methodology): “The LOA feature is calculated by Y = A ( x ) in which A performs audio to LOA encoding with a self-supervised representation learning module such as AudioMAE [21], [55].”