Text-Free Image-to-Speech Synthesis Using Learned Segmental Units

By Wei-Ning Hsu, David F. Harwath, Christopher Song, James R. Glass. 2020

75 citations recorded by the source corpus.

In this paper we present the first model for directly synthesizing fluent, natural-sounding spoken audio captions for images that does not require natural language text as an intermediate representation or source of supervision. Instead, we connect the image captioning module and the speech synthesis module with a set of discrete, sub-word speech units that are discovered with a self-supervised visual grounding task. We conduct experiments on the Flickr8k spoken caption dataset in addition to a novel corpus of spoken audio captions collected for the popular MSCOCO dataset, demonstrating that our generated captions also capture diverse visual semantics of the images they describe. We investigate several different intermediate speech representations, and empirically find that the representation must satisfy several important properties to serve as drop-in replacements for text.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Wei-Ning Hsu
2. Jiatao Gu
3. David F. Harwath
4. Luc Van Gool
5. Juan Pino
6. Jia-Bin Huang
7. Zhou Zhao
8. Yossi Adi

Citations in the corpus

• On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “7 We hypothesise that this simple modifica-tion allows to use Transformer’s limited attention span more efficiently as in Hsu et al. (2020).”
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, background): “…shown impressive results in the visually-induced audio generation (Su et al., 2020; Gan et al., 2020), towards generating realistic audio that describes the content of images or videos: Hsu et al. (2020) show that spoken language could be learned by a visually-grounded generative model of speech.”
• fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “…Hsu et al., 2021) to replace text for representing the lexical content in numerous applications, such as language modeling (Lakhotia et al., 2021), speech resynthesis (Polyak et al., 2021), image captioning (Hsu et al., 2020), and translation (Tjandra et al., 2020; Hayashi and Watanabe, 2020).”
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, background): “Since how to obtain speaker-invariant representation has been well-studied (Lee et al., 2021b; Hsu et al., 2020), we focus on the more challenging acoustic conditions in a single-speaker scenario, including rhythm, pitch, and energy variations.”
• Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, methodology): “DistilHuBERT (Chang, Yang, and Lee 2021) is a multitask knowledge distillation model built upon HuBERT (Hsu et al. 2021a) that drastically reduces the number of model parameters and inference time, while retains most of the performance on SUPERB.”