Spoken Content Retrieval—Beyond Cascading Speech Recognition with Text Retrieval

By Lin-Shan Lee, James R. Glass, Hung-yi Lee, Chun-an Chan. 2015

118 citations recorded by the source corpus.

Spoken content retrieval refers to directly indexing and retrieving spoken content based on the audio rather than text descriptions. This potentially eliminates the requirement of producing text descriptions for multimedia content for indexing and retrieval purposes, and is able to precisely locate the exact time the desired information appears in the multimedia. Spoken content retrieval has been very successfully achieved with the basic approach of cascading automatic speech recognition (ASR) with text information retrieval: after the spoken content is transcribed into text or lattice format, a text retrieval engine searches over the ASR output to find desired information. This framework works well when the ASR accuracy is relatively high, but becomes less adequate when more challenging real-world scenarios are considered, since retrieval performance depends heavily on ASR accuracy. This challenge leads to the emergence of another approach to spoken content retrieval: to go beyond the basic framework of cascading ASR with text retrieval in order to have retrieval performances that are less dependent on ASR accuracy. This overview article is intended to provide a thorough overview of the concepts, principles, approaches, and achievements of major technical contributions along this line of investigation. This includes five major directions: 1) Modified ASR for Retrieval Purposes: cascading ASR with text retrieval, but the ASR is modified or optimized for spoken content retrieval purposes; 2) Exploiting the Information not present in ASR outputs: to try to utilize the information in speech signals inevitably lost when transcribed into phonemes and words; 3) Directly Matching at the Acoustic Level without ASR: for spoken queries, the signals can be directly matched at the acoustic level, rather than at the phoneme or word levels, bypassing all ASR issues; 4) Semantic Retrieval of Spoken Content: trying to retrieve spoken content that is semantically related to the query, but not necessarily including the query terms themselves; 5) Interactive Retrieval and Efficient Presentation of the Retrieved Objects: with efficient presentation of the retrieved objects, an interactive retrieval process incorporating user actions may produce better retrieval results and user experiences.

Subfields: Artificial Intelligence, Information Retrieval & Search, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Speech & Audio

In Signal Processing

Citations in the corpus

FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, background): “As user interactions with machines move beyond textual queries, multi-modal retrieval of documents across the web is of growing interest [31, 32, 33].”
SpeechBERT: An Audio-and-Text Jointly Learned Language Model for End-to-End Spoken Question Answering (2019, background): “Various spoken language processing tasks, such as translation [1], retrieval [2], summarization [3] and understanding [4] have been very successful with a standard cascade architecture: an ASR front-end module transforming the speech signals into text form, followed by the downstream task module…”
On the Contributions of Visual and Textual Supervision in Low-resource Semantic Speech Retrieval (2019, unknown): “In semantic speech retrieval, the input is a textual query, a written word in our setting, and the task is to find utterances in a speech corpus that are semantically relevant to the query [14,15].”
MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible (2019, methodology): “We believe the obtained corpus can be useful in several applications, such as: speech-to-speech retrieval [2], multilingual speech representation learning [3] and direct speech-to-speech translation (so far, mostly direct speech-to-text translation has been investigated [4, 5, 6, 7]).”
Semantic Query-by-example Speech Search Using Visual Grounding (2019, background): “This differs from standard QbE, which only seeks exact matches to the query; from keyword spotting and spoken term detection, where the query is written instead of spoken; and from typical semantic search tasks [12,13], which also involve textual queries.”
Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech (2017, background): “There has been some work on semantic speech retrieval [76]–[79]; the most common approach is to cascade ASR with text-based retrieval.”