Loss Functions for Multiset Prediction

By Sean Welleck, Zi-Mei Yao, Yujie Gai, Jialin Mao, Zheng Zhang, Kyunghyun Cho. 2017

22 citations recorded by the source corpus.

We study the problem of multiset prediction. The goal of multiset prediction is to train a predictor that maps an input to a multiset consisting of multiple items. Unlike existing problems in supervised learning, such as classification, ranking and sequence generation, there is no known order among items in a target multiset, and each item in the multiset may appear more than once, making this problem extremely challenging. In this paper, we propose a novel multiset loss function by viewing this problem from the perspective of sequential decision making. The proposed multiset loss function is empirically evaluated on two families of datasets, one synthetic and the other real, with varying levels of difficulty, against various baseline loss functions including reinforcement learning, sequence, and aggregated distribution matching loss functions. The experiments reveal the effectiveness of the proposed loss function over the others.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, methodology): “[82] Sean Welleck, Zixin Yao, Yu Gai, Jialin Mao, Zheng Zhang, and Kyunghyun Cho.”
Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, background): “This is the case of [38, 39, 24], which model set cardinality as a categorical distribution, [24], which learns class-specific probability thresholds, and [52], which treats set prediction as a parameterized policy search problem.”
Information Leakage in Embedding Models (2020, neutral): “ependently. A more sophisticated formulation would be predicting the next word given the current predicted set of words until all words in the set are predicted. We adopt the multiset prediction loss [72] (MSP) that is suited for the formulation. Our MSP model trains a recurrent neural network that ”
NaturalProofs: Mathematical Theorem Proving in Natural Language (2021, background): “framing retrieval as multiset prediction (Welleck et al., 2018) – or recovering the order of references are also interesting to investigate.”
FSPool: Learning Set Representations with Featurewise Sort Pooling (2019, methodology): “Thus, the responsibility problem does not apply to methods for such problems, for example Welleck et al. (2018); Rezatoﬁghi. et al. (2017).”
Non-Monotonic Sequential Text Generation (2019, methodology): “However, recent work has found that, especially when dealing with recurrent neural network policies (which we will use; see § 4), using a cost function more analogous to a cross-entropy loss can be preferred (Leblond et al., 2018; Cheng et al., 2018; Welleck et al., 2018).”