Merging of the Senses
By Sidney S. Simon. 2008
1,354 citations recorded by the source corpus.
When we are constantly and simultaneously bombarded with various types of sensory inputs, how do we make decisions or motor outputs to deal with the world in a meaningful manner? This is a central issue in multisensory integration. One example of such type of processing is eating, as placing food in one's mouth provides taste, olfactory and somatosensory (texture, temperature) information (Verhagen and Engelen, 2006, Simon et al., 2006). Indeed, virtually all combinations of the senses are multimodal (Verhagen and Engelen, 2006). Whatever the combination of modalities, information from widely separated areas is rapidly integrated to give a motor response. In the case of taste the response to swallow or reject is yielded. One possibility that our brains have evolved to deal with multisensory stimuli could be due to the combined response reveals more about the nature of the external event. Our grand challenge is to know how information from different modalities that originally seem to be processed as unimodal events is combined during development to produce a coherent percept. This is known as the “binding problem” (Singer, 2001) and is important to all multisensory processing. Many issues regarding multisensory processing are found in a very recent and excellent review by Stein and Stanford (Stein and Stranford, 2008), and given the space limitations, I will not repeat them but rather give my opinion as to what is missing and what experiments would lead to a better understanding. It has been proposed that the entire neocortex is multisensory (Ghazanfar and Schroeder, 2006) and, although there has been some controversy about this proposal, it would be useful to have an atlas of the multisensory specificity of all subcortical and cortical areas. In electrophysiological studies of multisensory integration most recordings have been from single units, and confined to a single area. No doubt these recordings have lead to many advances and the formulation of rules that govern multisensory processing (e.g. superadditivity (Stein and Stranford, 2008), but it would be more informative if ensembles of neurons could be recorded in different areas while animals are being tested. This could be done either with implanted arrays of electrodes or with optical imaging (Nicolelis, 2007; Gradinaru, 2007). In this manner we will learn how different neurons in the same and different areas interact (synchronously or asynchronously) at different time scales. Such details can be use to determine which area signified the event (say congruent or non-congruent), even on a single trial (Nicolelis, 2007). We will also understand how the temporal structure of the neuronal dynamics provides information as to the behavioal task. The new technique involving the reversible use of light-induced activation or inactivation (Gradinaru, 2007) of neuronal ensembles in a given area will enable researchers to directly determine how the input of one modality affects the other modalitities and to determine whether multisensory information is feedforward and/or arises from top-down influences. This method may also permit researchers to begin to understand how animals determine saliency of cues to the extent as to which sensory modality yields most information. With these advances in animal research together with fMRI, EEG and evoked potential experiments in humans much progress will be made towards achieving our grand challenge – namely, to understand how the many facets of the sensory world around us becomes one coherent percept in our brain.
Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Sensory Systems, Multimodal Learning, Neuroscience
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cognitive Neuroscience
- Micah M. Murray
- Antonia Thelen
- Vincenzo Romei
- Charles Spence
- Yi-Chuan Chen
- Gregor Thut
- Céline Cappe
- Lorraine E. Bahrick
In Experimental and Cognitive Psychology
- Micah M. Murray
- Antonia Thelen
- David J. Lewkowicz
- Yi-Chuan Chen
- Charles Spence
- Vincenzo Romei
- Lorraine E. Bahrick
- Gregor Thut
In Sensory Systems
- Micah M. Murray
- Antonia Thelen
- Yi-Chuan Chen
- David J. Lewkowicz
- Charles Spence
- Vincenzo Romei
- Gregor Thut
- Céline Cappe
In Multimodal Learning
- Charles Spence
- Phil McAleer
- Franco Lepore
- David J. Lewkowicz
- Yi-Chuan Chen
- Lorraine E. Bahrick
- Colin Boudrias-Fournier
- Edmund C. Lalor
In Neuroscience
- Charles Spence
- Phil McAleer
- Franco Lepore
- David J. Lewkowicz
- Yi-Chuan Chen
- Lorraine E. Bahrick
- Colin Boudrias-Fournier
- Edmund C. Lalor
Citations in the corpus
- Assessing the Role of the ‘Unity Assumption’ on Multisensory Integration: A Review (2017, positive): “Specifically, multisensory stimuli that are presented close in time or space may have been encoded and/or integrated during the feed-forward processing (i.e., in a bottom–up fashion, see Stein and Meredith, 1993; Noesselt et al., 2010; van Atteveldt et al., 2014).”
- Multisensory Processes: A Balancing Act across the Lifespan (2016, neutral): “The convergence of inputs from multiple sensory modalities has been long known for classic regions of association cortex as well as for subcortical structures (reviewed in [6]).”
- Early Experience & Multisensory Perceptual Narrowing (2014, neutral): “…attributes are detected by our sensory systems, they are perceived as belonging to perceptually coherent and meaningful objects and events rather than as collections of unrelated sensations (Gibson, 1966; Maier & Schneirla, 1964; Marks, 1978; Ryan, 1940; Stein & Meredith, 1993; Werner, 1973).”
- Infant perception of audio-visual speech synchrony. (2010, positive): “This is highly advantageous because it increases perceptual salience and, as a result, enhances perception, learning, and discrimination (Bahrick, Lickliter, & Flom, 2004; Lewkowicz & Kraebel, 2004; Partan & Marler, 1999; Rowe, 1999; Stein & Meredith, 1993; Stein & Stanford, 2008).”
- Early experience determines how the senses will interact. (2007, neutral): “What appears critical for multisensory interactions is the close temporal and/or spatial correspondence of the different sensory signals derived from common events (Stein and Meredith 1993).”
- Modeling development of natural multi-sensory integration using neural self-organisation and probabilistic population codes (2015, neutral): “Natural organisms have therefore developed various independent sensory organs and ways to integrate their input (Stein & Meredith, 1993).”
- A Psychophysical Investigation of Differences between Synchrony and Temporal Order Judgments (2013, neutral): “One of the main techniques of multisensory integration research involves manipulating the temporal relationship of crossmodal cues to an event, and examining the consequences [1].”
- Eating with our ears: assessing the importance of the sounds of consumption on our perception and enjoyment of multisensory flavour experiences (2015, neutral): “the ventriloquist’s dummy and beeping flashing lights (see [72,73], for reviews).”
- The short-latency dopamine signal: a role in discovering novel actions? (2006, neutral): “Visual perception in the superior colliculus Reviews of visual processing in the mammalian superior colliculus agree that collicular neurons are exquisitely sensitive to spatially localized changes in luminance that signify appearance, disappearance or movement in the visual fiel”
- The Development of Cortical Multisensory Integration (2006, neutral): “The maturation of these behaviors has been well documented (Stein and Meredith, 1993) and follows a characteristic chronology with somatosensory responses appearing before birth (Stein et al., 1973), facilitating later orientation to the nipple (Larson and Stein, 1984).”
- Vestibular system: the many facets of a multimodal sense. (2008, neutral): “It is commonly thought that multisensory neural populations should represent different sensory signals in a common reference frame (Stein & Meredith 1993, Cohen & Andersen 2002).”
- Sound-Induced Enhancement of Low-Intensity Vision: Multisensory Influences on Human Sensory-Specific Cortices and Thalamic Bodies Relate to Perceptual Enhancement of Visual Detection Sensitivity (2010, neutral): “, 2008), and past proposals associated with the possible function of the putative principle of inverse effectiveness (Stein et al., 1988; Stein and Meredith, 1993; Kayser et al., 2008), we predicted that cooccurrence of a sound should benefit detection sensitivity (d ) for the lower-intensity but no”
- Crossmodal spatial attention (2010, neutral): “…participant happens to be performing another perceptually demanding task at the same time. c c However, returning to the neurophysiological data, 37 , 38 it is important to note that the multisensory superiority effect reported by Santangelo and Spence 67 , 68 should not necessarily be…”
- Oscillatory synchronization in large-scale cortical networks predicts perception. (2011, neutral): “…as unimodal as well as between early sensory and higher-order multimodal processing stages (Driver and Noesselt, 2008; Driver and Spence, 2000; Ghazanfar and Schroeder, 2006; Kayser et al., 2008; Lakatos et al., 2007; Lewis and Noppeney, 2010; Meredith et al., 2009; Stein and Meredith, 1993).”
- The impact of visual factors on noise annoyance among people living in the vicinity of wind turbines (2008, neutral): “…in respondents who lived in built-up areas (n ¼ 764) and respondents who lived in rural areas (n ¼ 331) through which we experience the environment (Stein & Meredith, 1993), and by using our cross-modal capabilities we greatly enhance the ability to recognize and detect external stimuli…”
- Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search? (2012, neutral): “Some basic principles have already been identified, such as the temporal and spatial rules (Meredith & Stein, 1986; Stein & Meredith, 1993).”