Diverse Beam Search for Improved Description of Complex Scenes

By Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, Dhruv Batra. 2018

289 citations recorded by the source corpus.

A single image captures the appearance and position of multiple entities in a scene as well as their complex interactions. As a consequence, natural language grounded in visual contexts tends to be diverse---with utterances differing as focus shifts to specific objects, interactions, or levels of detail. Recently, neural sequence models such as RNNs and LSTMs have been employed to produce visually-grounded language. Beam Search, the standard work-horse for decoding sequences from these models, is an approximate inference algorithm that decodes the top-B sequences in a greedy left-to-right fashion. In practice, the resulting sequences are often minor rewordings of a common utterance, failing to capture the multimodal nature of source images. To address this shortcoming, we propose Diverse Beam Search (DBS), a diversity promoting alternative to BS for approximate inference. DBS produces sequences that are significantly different from each other by incorporating diversity constraints within groups of candidate sequences during decoding; moreover, it achieves this with minimal computational or memory overhead. We demonstrate that our method improves both diversity and quality of decoded sequences over existing techniques on two visually-grounded language generation tasks---image captioning and visual question generation---particularly on complex scenes containing diverse visual content. We also show similar improvements at language-only machine translation tasks, highlighting the generality of our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “Similar to Li et al. [29], Vijayakumar et al. [53] and Massarelli et al. [35], we also investigate the diversity of generations by calculating the ratio of distinct ngrams to total ngrams generated by different models. diverse than RAG-Token generations, and both generate signiﬁcantly more diverse…”
The Curious Case of Neural Text Degeneration (2019, methodology): “Vijayakumar et al. (2018) proposed diverse beam search - a method for encouraging diversity in beam search-based generation which supports incorporating a task-specific diversity scoring function.”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “Other work has sought to explicitly promote diversity in the decoding process (Batra et al., 2012; Li et al., 2016; Vijayakumar et al., 2018).”
Neural Text Generation with Unlikelihood Training (2019, background): “Several forms of diverse beam search have been explored (Li et al., 2016; Vijayakumar et al., 2018; Kulikov et al., 2018; Holtzman et al., 2018) which can to some degree decrease the level of repetition of a model by selecting candidates unlike previously chosen ones, due to their variety.”
Importance of Search and Evaluation Strategies in Neural Dialogue Modeling (2018, methodology): “To address the locality issue in beam search, we propose an iterative beam search to radically increase the size of search space without introducing much computational overhead, inspired by earlier work on diverse beam search (Vijayakumar et al., 2018; Batra et al., 2012; Li et al., 2016).”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, neutral): “ranked siblings extended from the same parents, so that the N-best hypotheses in the beam at each time step are more likely to expand from different parents, and thus more diverse. Vijayakumar et al. [93] divided the hypotheses into several groups and applied beam search group by group. The model fa”
Mixture Models for Diverse Machine Translation: Tricks of the Trade (2019, background): “We also consider the following modified versions: Diverse Beam Search (Vijayakumar et al., 2018) and Biased Sampling (Graves, 2013; Fan et al.”
Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement (2019, methodology): “…have been developed, such as a continuous relaxation (Goyal et al., 2018), diversity encouraging variants (Li et al., 2016; Shao et al., 2017; Vijayakumar et al., 2018) or using modiﬁcations such as length-normalization (Wu et al., 2016) or simply applying noise to the output (Edunov et al.,…”
Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions (2018, background): “Some works have extended the beam-search algorithm to sample multiple captions from the same distribution [41, 1], while different GAN-based approaches have also appeared [8, 39, 45].”
Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020, methodology): “Diversity in text generation is often encouraged during inference time via heuristic modiﬁcations to beam search (Li et al., 2016; Vijayakumar et al., 2018), nucleus sampling (Holtzman et al., 2020) or submodular optimization (Kumar et al., 2019); in contrast, we simply ﬁlter our training data to…”
Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning (2019, methodology): “Moreover, we compare to beam search denoted as Beam search and diverse beam search [38] referred to as Div-BS applied to standard captioning methods based on convolutions [2] and LSTM nets [22].”
Diversifying Dialogue Generation with Non-Conversational Text (2020, background): “Detailed analysis are shown in § 6.3 to elaborate the differences among model performances and some case studies.”
Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots (2022, background): “Finally, researchers have dedicated a lot of effort to studying diverse decoding methods for better response generation [24, 29, 65].”
Analyzing Information Leakage of Updates to Natural Language Models (2019, background): “See [22] for more sophisticated techniques for increasing diversity.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “The most common technique to achieve diversity is based on variants of the beam search algorithm [213] that”
TextBox: A Unified, Modularized, and Extensible Framework for Text Generation (2021, background): “ties for anything below the k-th token, and beam search (Vijayakumar et al., 2018) strategy selects the top scoring B candidates from the set of all pos-”
BRIO: Bringing Order to Abstractive Summarization (2022, methodology): “This is achieved by using diverse beam search (Vijayakumar et al., 2018).”
Mixture Content Selection for Diverse Sequence Generation (2019, methodology): “3, (b) refers to search-based methods (Vijayakumar et al., 2018; Li et al., 2016b; Fan et al., 2018), and (c) refers to mixture decoders (Shen et al.”
LightSeq: A High Performance Inference Library for Transformers (2021, methodology): “Further, LightSeq provides different search algorithms, such as beam search, diverse beam search and probabilistic sampling (Vijayakumar et al., 2018).”