Object Familiarity and Semantic Congruency Modulate Responses in Cortical Audiovisual Integration Areas

By Grit Hein, O. Doehrmann, Notger G. Müller, Jochen Kaiser, Lars Muckli, Marcus J. Naumer. 2007

217 citations recorded by the source corpus.

The cortical integration of auditory and visual features is crucial for efficient object recognition. Previous studies have shown that audiovisual (AV) integration is affected by where and when auditory and visual features occur. However, because relatively little is known about the impact of what is integrated, we here investigated the impact of semantic congruency and object familiarity on the neural correlates of AV integration. We used functional magnetic resonance imaging to identify regions involved in the integration of both (congruent and incongruent) familiar animal sounds and images and of arbitrary combinations of unfamiliar artificial sounds and object images. Unfamiliar object images and sounds were integrated in the inferior frontal cortex (IFC), possibly reflecting learning of novel AV associations. Integration of familiar, but semantically incongruent combinations also correlated with IFC activation and additionally involved the posterior superior temporal sulcus (pSTS). For highly familiar semantically congruent AV pairings, we again found AV integration effects in pSTS and additionally in superior temporal gyrus. These findings demonstrate that the neural correlates of object-related AV integration reflect both semantic congruency and familiarity of the integrated sounds and images.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Sensory Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

In Sensory Systems

Citations in the corpus

Assessing the Role of the ‘Unity Assumption’ on Multisensory Integration: A Review (2017, background): “Researchers often presume that visual and auditory stimuli are integrated when they are semantically congruent and some form of enhancement in their participants’ behavioral performance or brain activities will be observed (Laurienti et al., 2004; Molholm et al., 2004; Taylor et al., 2006; Hein et a”
Neural reflections of meaning in gesture, language, and action (2009, background): “LIFG was found to be sensitive to the amount of semantic congruity of simultaneously presented picture of an animal and the sound of the animal (Hein et al. 2007).”
Responses of Prefrontal Multisensory Neurons to Mismatching Faces and Vocalizations (2014, background): “, 2013), several studies using incongruent AV stimuli have observed changes in the activity of the inferior frontal gyrus (IFG; Ojanen et al., 2005; Hein et al., 2007; Noppeney et al., 2008; Naumer et al., 2009), a human brain region similar to the macaque VLPFC (Petrides and Pandya, 2002; Romanski,”
Superior Temporal SulcusIt's My Area: Or Is It? (2008, background): “…versus fixation 51, 36, 9* Calvert, Hansen, Iversen, and Brammer (2001) Pictures and sounds of common objects versus fixation 46, 76, 22* Taylor, Moss, Stamatakis, and Tyler (2006) Pictures and sound of common and abstract objects versus fixation 44, 43, 15 55, 29, 15** Hein et al. (2007)”
Multisensory visual–tactile object related network in humans: insights gained using a novel crossmodal adaptation approach (2009, methodology): “For instance, Hein and colleagues found multisensory integration and congruency eVects for auditory and visual objects in the neighboring (probably more ventral) prefrontal cortex (Hein et al. 2007).”
Measuring structural-functional correspondence: Spatial variability of specialised brain regions after macro-anatomical alignment (2012, unknown): “…that animal makes, “incongruent” where an animal image was paired with the sound of a different animal, “images alone”where the imageswere presented without any accompanying auditory input and “sounds alone” where the animal noiseswere presented in the absence of visual stimuli (Hein et al., 2007).”
Prior auditory information shapes visual category-selectivity in ventral occipito-temporal cortex (2010, unknown): “, 2007) and higher order executive regions (Hein et al., 2007; Noppeney et al., 2008; Olivetti Belardinelli et al., 2004; for review see Doehrmann and Naumer, 2008) in AV (in)congruency.”
The effect of synesthetic associations between the visual and auditory modalities on the Colavita effect (2015, background): “Neuroimaging and electrophysiological studies also found semantic influences on the neural correlates of audiovisual integration (Molholm et al. 2004; Hein et al. 2007; Stekelenburg and Vroomen 2007; Yuval-Greenberg and Deouell 2007).”
Distinct Functional Contributions of Primary Sensory and Association Areas to Audiovisual Integration in Object Categorization (2010, background): “, 2008) and semantic (what?) (Hein et al., 2007; Noppeney et al., 2008) congruency of signals from different senses.”
Distinct Computational Principles Govern Multisensory Integration in Primary Sensory and Association Cortices. (2016, unknown): “Accumulating evidence has demonstrated that multisensory integration is not deferred to association cortices [17–22] but starts already at the primary, putatively unisensory level [23–28] via thalamo-cortical mechanisms [27], direct connectiv-”
How prior expectations shape multisensory perception (2016, background): “…showing activation increases for a variety of stimuli (e.g. speech, objects) that are characterized by different types of incongruent relationships (Hein et al., 2007; Naumer et al., 2009; Noppeney et al., 2010; Ojanen et al., 2005; Olivetti Belardinelli et al., 2004; Pekkola et al., 2006; van…”