Mixture Density Networks

By Sargur N. Srihari. 1994

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Generating Sequences With Recurrent Neural Networks (2013, methodology): “The idea of mixture density networks [2, 3] is to use the outputs of a neural network to parameterise a mixture distribution.”
WaveNet: A Generative Model for Raw Audio (2016, methodology): “One approach to modeling the conditional distributions p ( x t | x 1 , . . . , x t − 1 ) over the individual audio samples would be to use a mixture model such as a mixture density network (Bishop, 1994) or mixture of conditional Gaussian scale mixtures (MCGSM) (Theis & Bethge, 2015).”
Symbolic Music Generation with Diffusion Models (2021, methodology): “We compare our model to an autoregressive transformer with a mixture density output layer [22] and train on the same dataset as the diffusion model.”
f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016, background): “Mixture density networks [2] are neural networks which directly regress the parameters of a finite parametric mixture model.”
RNADE: The real-valued neural autoregressive density-estimator (2013, methodology): “That is, the outputs are mixture density networks [6], with a shared hidden layer, using the same parameter tying as NADE.”
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions (2017, methodology): “We also experimented with a log-likelihood loss by modeling the output distribution with a Mixture Density Network [23, 24] to avoid assuming a constant variance over time, but found that these were more difficult to train and they did not lead to better sounding samples.”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, background): “(2) In cases where the Gaussian is too-restrictive, one could use a complex distribution e.g. mixture density network (Bishop, 1994) or a heavy-tailed distribution.”
Variational Information Distillation for Knowledge Transfer (2019, methodology): “Other types of parameterizations such as a heavy-tailed distribution or the mixture density network [5] could be used to gain additional performance.”
Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, methodology): “For example, the deterministic dynamics model has been extended by parameterizing it with a Gaussian distribution or a Gaussian mixture distribution for better generalization [28].”
A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms (2019, methodology): “We model the conditionals with mixture density networks (Bishop, 1994) and the marginals by a Gaussian mixture.”
World Models (2018, methodology): “This approach is known as a Mixture Density Network (Bishop, 1994) combined with a RNN (MDN-RNN) (Graves, 2013; Ha, 2017a), and has been applied in the past for sequence generation problems such as generating handwriting (Graves, 2013) and sketches (Ha & Eck, 2017).”
A Multi-Horizon Quantile Recurrent Forecaster (2017, background): “n identical encoder and decoder. DeepAR directly outputs parameters of a Negative Binomial. This is similar toNg et al, 2017where an MLP predicts Gaussian parameters, and such a strategy dates back toBishop, 1994. DeepAR is trained by maximizing likelihood and Teacher Forcing (feeding ground truth r”
Implicit Behavioral Cloning (2021, methodology): “Explicit “MSE” models are 60 trained with Mean Square Error (MSE), explicit “MDN” models are Mixture Density Networks (MDN) 61 [21], and implicit “EBM” models are trained with LInfoNCE and optimized with derivative-free optimization.”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, methodology): “Behavior policy We ﬁt the behavior model using a conditional Mixture of Gaussians (Bishop, 1994) with tanh squash-ing (Haarnoja et al., 2017).”
Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, methodology): “To represent multi-modality, ConvMLP, GT-State MLP, and 2-step GT-State MLP regress mixture densities [43].”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, methodology): “At the output, we use mixture density networks (MDN) [2] to represent the distribution over actions.”
One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning (2018, methodology): “All methods use a mixture density network [4] to represent the action space, where the actions correspond to the linear and rotational velocity of the robot gripper, a 6-dimensional continuous action space.”
Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-to-End Learning from Demonstration (2017, methodology): “While autoregressive estimators usually discretize the output, we use mixture of Gaussians to predict the entire probability distribution of the output, providing a rich and expressive class of distributions.”
Detecting and Recognizing Human-Object Interactions (2017, methodology): “To improve target localization prediction, we tried to substitute the uni-modal regression network with a Mixture Density Network (MDN) [1].”