Mixture Content Selection for Diverse Sequence Generation

By Jaemin Cho, Minjoon Seo, Hannaneh Hajishirzi. 2019

64 citations recorded by the source corpus.

Generating diverse sequences is important in many NLP applications such as question generation or summarization that exhibit semantically one-to-many relationships between source and the target sequences. We present a method to explicitly separate diversification from generation using a general plug-and-play module (called SELECTOR) that wraps around and guides an existing encoder-decoder model. The diversification stage uses a mixture of experts to sample different binary masks on the source sequence for diverse content selection. The generation stage uses a standard encoder-decoder model given each selected content from the source sequence. Due to the non-differentiable nature of discrete sampling and the lack of ground truth labels for binary mask, we leverage a proxy for ground truth mask and adopt stochastic hard-EM for training. In question generation (SQuAD) and abstractive summarization (CNN-DM), our method demonstrates significant improvements in accuracy, diversity and training efficiency, including state-of-the-art top-1 accuracy in both datasets, 6% gain in top-5 accuracy, and 3.7 times faster training over a state-of-the-art model. Our code is publicly available at https://github.com/clovaai/FocusSeq2Seq.

Subfields: Artificial Intelligence, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Generative Models

Citations in the corpus

How Decoding Strategies Affect the Verifiability of Generated Text (2019, background): “Cho et al. (2019) choose to separate the generation and diversification steps altogether, and focus on leveraging content selection to map the input to diverse sequences.”
Unsupervised Quality Estimation for Neural Machine Translation (2020, methodology): “First , we consider three system variants with differences in architecture or training: RNN-based NMT (Bahdanau et al., 2015; Luong et al., 2015), Mixture of Experts (MoE, He et al., 2018; Shen et al., 2019; Cho et al., 2019) and model ensemble (Garmash and Monz, 2016).”
Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts (2022, methodology): “This research problem is also referred as one-to-many generation (Shen et al., 2019; Cho et al., 2019; Yu et al., 2021; Shen et al., 2022).”
Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021, methodology): “Following Gehrmann et al. (2018), Cho et al. (2019) use a mixture of experts to sample different binary masks on the source sequence for diverse content selection for summarization.”