Linguistic Unit Discovery from Multi-Modal Inputs in Unwritten Languages: Summary of the “Speaking Rosetta” JSALT 2017 Workshop

By O. Scharenborg, Laurent Besacier, Alan W. Black, Mark Hasegawa-Johnson, Florian Metze, Graham Neubig, Sebastian Stüker, Pierre Godard, Markus Müller, Lucas Ondel and 9 more. 2018

34 citations recorded by the source corpus.

We summarize the accomplishments of a multi-disciplinary workshop exploring the computational and scientific issues surrounding the discovery of linguistic units (subwords and words) in a language without orthography. We study the replacement of orthographic transcriptions by images and/or translated text in a well-resourced language to help unsupervised discovery from raw speech.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning, Linguistics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

In Linguistics

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “…to the modeling or training algorithms (Leidal et al., 2017; Kamper et al., 2017c; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Scharenborg et al., 2018; Kamper et al., 2019b;a; Surı́s et al., 2019; Ilharco et al., 2019; Eloff et al., 2019a), application to multilingual…”
Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks (2019, background): “But, importantly, the task here involves unit discovery which compresses spoken input [23,24].”
The Zero Resource Speech Challenge 2019: TTS without T (2019, background): “A limited number of papers have provided a proof of concept that TTS without T is feasible [3, 4].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “Most closely related to this paper is the work combining paired image and speech information in an unsupervised setting [1, 2, 8–17].”
Self-Supervised Speech Representation Learning: A Review (2022, background): “other contextual modality), such models often learn multiple levels of linguistic representations from the shallowest to the deepest model layers [181], [182], [183].”
Effectiveness of Self-Supervised Pre-Training for ASR (2020, background): “The practical need for building ASR systems for new conditions with limited resources spurred a lot of work focused on unsupervised speech recognition and representation learning [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], in addition to semi- and weakly-supervised learning techniques to reduce th”
On the Contributions of Visual and Textual Supervision in Low-resource Semantic Speech Retrieval (2019, unknown): “Recent work has begun exploring alternative sources of weak supervision, such as visual grounding from images [1–7].”
Towards Visually Grounded Sub-word Speech Unit Discovery (2019, background): “Subsequent works have continued to explore the leveraging of visual information to guide models of speech audio data [15, 16, 17, 18, 19, 20], and several papers have begun to investigate the nature of the internal representations learned by these visually-grounded models [21, 22].”
Cyclic Spectral Modeling for Unsupervised Unit Discovery into Voice Conversion with Excitation and Waveform Modeling (2020, background): “Recently, a lot of work have been carried out on the so-called zero resource settings of speech processing [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”