Compositional Visual Generation with Energy Based Models

By Yilun Du, Shuang Li, Igor Mordatch. 2020

217 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, background): “Table III shows that CLIP excels with descriptive queries but struggles with complex affordance and negation queries [62].”
Improved Contrastive Divergence Training of Energy Based Models (2020, background): “Energy-Based Models (EBMs) have the ability to compose with other models at generation time (Hinton, 1999; Haarnoja et al., 2017; Du et al., 2020a).”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “One such method learns one energy-based model per attribute that can later be combined with other attributes in novel combinations—for example, to generate a smiling man from the attributes “smiling” and “man” (Du et al., 2020).”
Improving Factuality and Reasoning in Language Models through Multiagent Debate (2023, background): “State your final answer in a newline with a 2 letter response following the regex [a-h][1-8].”
Diffusion models as plug-and-play priors (2022, background): “3 we showcase the intermediate steps of the optimization process for inference with the conditions blond+smiling+not male, thus solving a problem like that studied in [8] using only independently trained attribute classifiers and an unconditional generative model of faces.”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, methodology): “By composing energy functions with logical operators [12], our method is the first to show such compositionality in generating photo-realistic images of resolution 1024⇥1024.”
Composing Ensembles of Pre-trained Models via Iterative Consensus (2022, methodology): “Other works compose pre-trained models by composing learned probability distributions of the data, such as energy-based models (Liu et al., 2022; 2021; Du et al., 2020), which can be applied to image generation.”
Compositional Visual Generation with Composable Diffusion Models (2022, methodology): “Thus, most relevant to our work, existing work has utilized EBMs to compose different factors describing a scene [8, 10, 23, 29].”
Planning with Diffusion for Flexible Behavior Synthesis (2022, background): “Iterative, gradient-based sampling lends itself towards ﬂexible conditioning (Dhariwal & Nichol, 2021) and compositionality (Du et al., 2020b), which we use to recover effective behaviors from heterogeneous datasets and plan for reward functions unseen during training.”
Erasing Concepts from Diffusion Models (2023, background): “Our work is inspired by the observation [10, 11] that set-like composition can be performed naturally on energy-based models and diffusion counterparts [22] naturally via arithmetic on the score or the noise predictions.”