VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation

By Tianrui Wang, Long Zhou, Zi-Hua Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, Furu Wei. 2023

123 citations recorded by the source corpus.

Recent research shows a big convergence in model architecture, training objectives, and inference methods across various tasks for different modalities. In this paper, we propose VioLA, a single auto-regressive Transformer decoder-only network that unifies various cross-modal tasks involving speech and text, such as speech-to-text, text-to-text, text-to-speech, and speech-to-speech tasks, as a conditional codec language model task via multi-task learning framework. To accomplish this, we first convert all the speech utterances to discrete tokens (similar to the textual data) using an offline neural codec encoder. In such a way, all these tasks are converted to token-based sequence conversion problems, which can be naturally handled with one conditional language model. We further integrate task IDs (TID) and language IDs (LID) into the proposed model to enhance the modeling capability of handling different languages and tasks. Experimental results demonstrate that the proposed VioLA model can support both single-modal and cross-modal tasks well, and the decoder-only model achieves a comparable and even better performance than the strong baselines.

Subfields: Artificial Intelligence, Large Language Models, Machine Translation, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Machine Translation

In Speech & Audio

In Signal Processing

Citations in the corpus

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “In addition to speech generation, VioLA[72]exploredthepotentialofcodec-basedspeechmodels forspeechunderstandingtasks,aimingtounifycodeclanguage modelsforspeechrecognition,synthesis,andtranslationtasks.”
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “Inspired by the success of Large Language Models (LLMs) in natural language processing, VALL-E [Wang et al., 2023a, Zhang et al., 2023b] represented speech as discrete codec codes with an off-the-shelf neural codec model, and approached TTS as a conditional codec language modeling task.”
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “From BASE TTS’s strong performance on English and Spanish, we caught a first glimpse of a multilingual TTS approach that achieves high expressiveness, adapts to textual clues, is data efficient, uses only public domain data, and works for streaming TTS usecases such as voicing LLM outputs.”
FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec (2023, background): “Moreover, neural speech codecs facilitate the modeling of speech and text within a single framework, enabling the model to both listen and speak [17,18].”
RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, background): “Besides, recent works [22, 30, 33] have shown the decoder-only architecture can be used to learn multiple tasks, as the input and output are processed jointly by a language model, and the model can be signaled to generate results for different tasks by inputting pre-defined special tokens.”
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models (2023, background): “Many works (Wang et al., 2023d; Maiti et al., 2023; Rubenstein et al.; Wang et al., 2023e; Nachmani et al., 2023) unify data format and tasks by directly feeding speech representation (Nachmani et al., 2023) or encoding continuous speech signals as discrete codes (Défossez et al., 2022; Zeghidour…”
Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognition (2023, methodology): “One class of foundational audio model efforts like VALL-E [8], VALL-E X [9], VIOLA [10], MusicLM [11] use compression based tokens where as models such as AudioLM [12], AudioPaLM [13] employ semantic tokens.”
Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS (2023, background): “Recently, Wang et al. [17] presented VioLA, transforming speech signals into discrete tokens, enabling a unified conditional language modeling way for multiple spoken processing tasks.”
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, methodology): “The fourth group, such as VioLA (Wang et al., 2023b), AudioPaLM (Rubenstein et al., 2023), SpeechGPT (Zhang et al., 2023a) and SpeechGen (Wu et al., 2023), use decoder-only Transformers to model discrete audio tokens and text tokens as a shared vocabulary, but they may suffer from the information lo”
VoxtLM: Unified Decoder-Only Models for Consolidating Speech Recognition, Synthesis and Speech, Text Continuation Tasks (2023, background): “PolyVoice [18] applies speechLM to speech-to-speech translation (S2ST) with three decoder-only LMs. VioLA [19] extends VALL-E [7] for ASR and S2ST.”
F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, background): “Recent research in Text-to-Speech (TTS) has experienced great advancement [1, 2, 3, 4, 5, 6, 7, 8].”
End-to-End Speech Recognition: A Survey (2023, background): “In addition, in line with the growing interest in large language models in recent years, research has also begun on solving multiple tasks, including speech recognition, using only an auto-regressive, GPT-style decoder [110], [111].”