Is Someone Speaking?: Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection

By Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, Mike Zheng Shou, Haizhou Li. 2021

256 citations recorded by the source corpus.

Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and long-term audio and visual information, as well as audio-visual interaction. Unlike the prior work where systems make decision instantaneously using short-term features, we propose a novel framework, named TalkNet, that makes decision by taking both short-term and long-term features into consideration. TalkNet consists of audio and visual temporal encoders for feature representation, audio-visual cross-attention mechanism for inter-modality interaction, and a self-attention mechanism to capture long-term speaking evidence. The experiments demonstrate that TalkNet achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been made available at: https://github.com/TaoRuijie/TalkNet_ASD.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Kristen Grauman
  2. Trevor Darrell
  3. Giovanni Maria Farinella
  4. Antonino Furnari
  5. Abhinav Kumar Gupta
  6. Luc Van Gool
  7. Jitendra Malik
  8. Christoph Feichtenhofer

In Signal Processing

  1. Kristen Grauman
  2. Jitendra Malik
  3. Lorenzo Torresani
  4. Antonio Torralba
  5. David J. Crandall
  6. Christoph Feichtenhofer
  7. Tushar Nagarajan
  8. Rohit Girdhar

Citations in the corpus