Underspecification in Scene Description-to-Depiction Tasks

By Ben Hutchinson, Jason Baldridge, Vinodkumar Prabhakaran. 2022

43 citations recorded by the source corpus.

Questions regarding implicitness, ambiguity and underspecification are crucial for understanding the task validity and ethical concerns of multimodal image+text systems, yet have received little attention to date. This position paper maps out a conceptual framework to address this gap, focusing on systems which generate images depicting scenes from scene descriptions. In doing so, we account for how texts and images convey meaning differently. We outline a set of core challenges concerning textual and visual ambiguity, as well as risks that may be amplified by ambiguous and underspecified elements. We propose and discuss strategies for addressing these challenges, including generating visually ambiguous images, and generating a set of diverse images.

Subfields: Computer Vision and Pattern Recognition, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Generation

Citations in the corpus

Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, background): “, 2022)), or conceptual vagueness around constructs in the task formulation (Hutchinson et al., 2022).”
PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, background): “Large models, if left unchecked, have the potential to inflict harm on society – such as amplifying biases [76, 77, 78, 79], causing disparities [78, 80, 81], or encoding narrow cultural perspectives [82, 83].”