The Natural Statistics of Audiovisual Speech

By Chandramouli Chandrasekaran, Andrea Trubanova, Sébastien Stillittano, Alice Caplier, Asif A. Ghazanfar. 2009

622 citations recorded by the source corpus.

Humans, like other animals, are exposed to a continuous stream of signals, which are dynamic, multimodal, extended, and time varying in nature. This complex input space must be transduced and sampled by our sensory systems and transmitted to the brain where it can guide the selection of appropriate actions. To simplify this process, it's been suggested that the brain exploits statistical regularities in the stimulus space. Tests of this idea have largely been confined to unimodal signals and natural scenes. One important class of multisensory signals for which a quantitative input space characterization is unavailable is human speech. We do not understand what signals our brain has to actively piece together from an audiovisual speech stream to arrive at a percept versus what is already embedded in the signal structure of the stream itself. In essence, we do not have a clear understanding of the natural statistics of audiovisual speech. In the present study, we identified the following major statistical features of audiovisual speech. First, we observed robust correlations and close temporal correspondence between the area of the mouth opening and the acoustic envelope. Second, we found the strongest correlation between the area of the mouth opening and vocal tract resonances. Third, we observed that both area of the mouth opening and the voice envelope are temporally modulated in the 2–7 Hz frequency range. Finally, we show that the timing of mouth movements relative to the onset of the voice is consistently between 100 and 300 ms. We interpret these data in the context of recent neural theories of speech which suggest that speech communication is a reciprocally coupled, multisensory event, whereby the outputs of the signaler are matched to the neural processes of the receiver.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

Citations in the corpus

Early Experience & Multisensory Perceptual Narrowing (2014, background): “First, the speaker’s mouth is where the tightly coupled and highly redundant patterns of auditory and visual speech information that imbue audiovisual speech with its greater salience are located (Chandrasekaran et al., 2009; Munhall & Vatikiotis-Bateson, 2004).”
A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula (2016, methodology): “entrainment mostly at lower frequencies, we extracted the wideband amplitude envelope of the speech stimulus [Chandrasekaran et al., 2009; Gross et al., 2013] and then low-pass filtered with a 12 Hz cutoff (third order noncausal Butterworth).”
Multisensory integration in auditory and auditory-related areas of cortex (2009, result): “A catalog of typical or normative lags between lip movements and sound onset recently constructed by Ghazanfar and colleagues (Chandrasekaran et al., 2009), indicates that, in agreement with earlier more anectodal reports (Schroeder et al.”
Lip Movements Generation at a Glance (2018, background): “For instance, when we say word ‘bed’, upper and lower lips meet before speaking the word [2].”
Combining Residual Networks with LSTMs for Lipreading (2017, neutral): “Prior to the advent of deep learning ([10]) most of the work in lipreading was based on hand-engineered features, that were usually modeled by HMM-based pipeline, [11] [12] [13] [14] [15].”
Chronux: A Platform for Analyzing Neural Signals (2010, methodology): “7), we compute spike-field coherence (van der Meer and Redish, 2009;Chandrasekaran et al., 2009) for data recorded from the primary visual cortex of a monkey during an attention modulation task (Womelsdorf et al., 2006).”
The COGs (context, object, and goals) in multisensory processing (2016, background): “…both within and across the senses (reviewed in Frost et al. 2015), with effects transpiring across multiple temporal scales (e.g. Baier et al. 2006; Beierholm et al. 2009; Chandrasekaran et al. 2009; Barakat et al. 2013; Barenholtz et al. 2014; Altieri et al. 2015; Sarmiento et al. 2012, 2016).”
Multisensory integration: flexible use of general operations (2014, neutral): “Visual cues fine-tune the entrainment to the attended speech stream in the auditory cortex (Figure 3C), and this crossmodal phase reset is predictive (Figure 3D); as facial articulatory cues and head movements precede the auditory speech input (Chandrasekaran et al., 2009), they provide predictions ”