SDR – Half-baked or Well Done?
By Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, John R. Hershey. 2018
1,662 citations recorded by the source corpus.
In speech enhancement and source separation, signal-to-noise ratio is a ubiquitous objective measure of denoising/separation quality. A decade ago, the BSS_eval toolkit was developed to give researchers worldwide a way to evaluate the quality of their algorithms in a simple, fair, and hopefully insightful way: it attempted to account for channel variations, and to not only evaluate the total distortion in the estimated signal but also split it in terms of various factors such as remaining interference, newly added artifacts, and channel errors. In recent years, hundreds of papers have been relying on this toolkit to evaluate their proposed methods and compare them to previous works, often arguing that differences on the order of 0.1 dB proved the effectiveness of a method over others. We argue here that the signal-to-distortion ratio (SDR) implemented in the BSS_eval toolkit has generally been improperly used and abused, especially in the case of single-channel separation, resulting in misleading results. We propose to use a slightly modified definition, resulting in a simpler, more robust measure, called scale-invariant SDR (SI-SDR). We present various examples of critical failure of the original SDR that SI-SDR overcomes.
Subfields: Cognitive Neuroscience, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cognitive Neuroscience
- Deliang Wang
- Ke Tan
- John R. Hershey
- Yi Luo
- Jonathan Le Roux
- Nima Mesgarani
- Zhong-Qiu Wang
- Yuxuan Wang
In Speech & Audio
In Signal Processing
- Yossi Adi
- Jinyu Li
- Nima Mesgarani
- Deliang Wang
- Shinji Watanabe
- John R. Hershey
- Zhuo Chen
- Jonathan Le Roux
Citations in the corpus
- High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “First, we find that the multi-band STFT discriminator does not result in significantly better metrics, except for SI-SDR, where it is slightly better.”
- Real-Time Speech Frequency Bandwidth Extension (2020, methodology): “Two objective metrics are used to measure the reconstruction quality of the reconstructed audio with respect to the ground truth: • Scale-invariant signal-to-distortion ratio (SI-SDR) [23] over the audio waveform, which measures the per-sample fidelity up to a uniform scaling factor.”
- Voice Separation with an Unknown Number of Multiple Speakers (2020, background): “However, recent studies have argued that the above mentioned metric has been improperly used due to its filter dependence and may result in misleading findings (Le Roux et al., 2019).”
- NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality (2022, background): “1) Judgment Guideline: While there are some objective metrics to measure the quality gap between the generated speech and human recordings, such as PESQ [30], STOI [31], SI-SDR [32], they are not reliable to measure the perception quality in TTS [33], [34].”
- End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, methodology): “We report the performance using scale-invariant SDR (SISDR) [1, 2, 5, 39], as well as the SDR metric computed using the bss eval sources software [40] because it is used by other groups.”
- Recursive speech separation for unknown number of speakers (2019, methodology): “econd output channel until the stopping criterion is met. One notable advantage of the proposed method is that it is not required to predefine the maximum number of speakers 1Also denoted as SI-SDR in [12,23]. and can be applied to an arbitrary number of sources even those never seen during the train”
- Phase-aware Single-stage Speech Denoising and Dereverberation with U-Net (2020, methodology): “To show the effect of loss functions we observed SI-SDR [10] and PESQ [29] while changing four different loss functions.”
- Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms (2018, background): “Le Roux et al. [11] have recently brought to light some weaknesses of SDR and this implementation in particular.”
- Generative Pre-training for Speech with Flow Matching (2023, methodology): “We report the common metric Scale-Invariant Signal-to-Distortion Ratio improvement (SI-SDRi; Le Roux et al., 2019) that measures the improvement of separated speech over the mixture when comparing against the clean reference in the time domain.”
- Filterbank Design for End-to-end Speech Separation (2019, neutral): “Training is performed on 4 s segments using the permutationinvariant [2,4] scale-invarariant source-to-distorsion ratio (SISDR) [9, 19] as the training objective.”
- Attention Is All You Need In Speech Separation (2020, neutral): “Our model is trained with batches composed of single sentences by maximizing the Scale-Invariant Signal-to-Noise Ratio (SI-SNR) [27] with utterance-level permutation invariant loss [13].”
- FurcaNeXt: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks (2019, background): “For further work, although SDR is widely used and can be useful, but it has some weaknesses [19].”
- Performance Study of a Convolutional Time-Domain Audio Separation Network for Real-Time Speech Denoising (2020, methodology): “It should be noted that SI-SAR was not considered in the original Tas-Net papers [10, 11], and that the usefulness of separating out SAR from SDR has been questioned [19].”
- Torchaudio-Squim: Reference-Less Speech Quality and Intelligibility Measures in Torchaudio (2023, methodology): “These are Perceptual Evaluation of Speech Quality (PESQ) [19, 20], Short-Time Objective Intelligibility (STOI) [21] and Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) [22].”