Learning to integrate arbitrary signals from vision and touch.

By Marc O. Ernst. 2007

340 citations recorded by the source corpus.

When different perceptual signals of the same physical property are integrated, for example, an objects' size, which can be seen and felt, they form a more reliable sensory estimate (e.g., M. O. Ernst & M. S. Banks, 2002). This, however, implies that the sensory system already knows which signals belong together and how they relate. In other words, the system has to know the mapping between the signals. In a Bayesian model of cue integration, this prior knowledge can be made explicit. Here, we ask whether such a mapping between two arbitrary sensory signals from vision and touch can be learned from their statistical co-occurrence such that they become integrated. In the Bayesian framework, this means changing the belief about the distribution of the stimuli. To this end, we trained subjects with stimuli that are usually unrelated in the world--the luminance of an object (visual signal) and its stiffness (haptic signal). In the training phase, we then presented subjects with combinations of these two signals, which were artificially correlated, and thus, we introduced a new mapping between them. For example, the stiffer the object, the brighter it was. We measured the influence of learning by comparing discrimination performance before and after training. The prediction is that integration makes discrimination worse for stimuli, which are incongruent with the newly learned mapping, because integration would cause this incongruency to disappear perceptually. The more certain subjects are about the new mapping, the stronger should the influence be on discrimination performance. Thus, learning in this context is about acquiring beliefs. We found a significant change in discrimination performance before and after training when comparing trials with congruent and incongruent stimuli. After training, discrimination thresholds for the incongruent stimuli are increased relative to thresholds for congruent stimuli, suggesting that subjects learned effectively to integrate the two formerly unrelated signals.

Subfields: Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety Research

Citations in the corpus

Assessing the Role of the ‘Unity Assumption’ on Multisensory Integration: A Review (2017, methodology): “In one of the most convincing studies using the prior term in a Bayesian model to represent the learning of the mapping between crossmodal signals, Ernst (2007) trained the participants with typically unrelated visual and tactile features; specifically, luminance and stiffness.”
Cross-correlation between auditory and visual signals promotes multisensory integration. (2013, background): “Recently, several implementations of Bayesian models have been proposed in order to describe the effects of causal inference in multisensory integration (Ernst, 2007; Ernst and Di Luca, 2011; Knill, 2007; Körding et al., 2007).”
‘When Birds of a Feather Flock Together’: Synesthetic Correspondences Modulate Audiovisual Integration in Non-Synesthetes (2009, background): “, the smaller the variance of the coupling prior), the stronger such stimuli will be coupled [19,30].”
The Extraction of Neural Information from the Surface EMG for the Control of Upper-Limb Prostheses: Emerging Avenues and Challenges (2014, background): “It also underlines the importance, which has been recently recognized for most human–machine interaction systems, of testing the performance of the system based on online experiments in which the user is within the loop and can adapt to the control system [85]–[87].”
Bayesian priors are encoded independently from likelihoods in human multisensory perception. (2009, background): “…expectation of a common cause is expected to be mostly due to the learned or hard - wired statistics of the auditory - visual events in the environment , it may also be affected by the instructions provided to the observer by the experimenter or the context of the experiment ( Ernst , 2007 ) .”
Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search? (2012, background): “…between auditory pitch and visual brightness were to be semantic in nature, it could possibly never manifest itself at a preattentive stage of perceptual processing (see also Gallace & Spence, 2006), while a structural or statistical correspondence might (see Ernst, 2007; Parise & Spence, 2009).”
Crossmodal correspondences: A tutorial review (2011, background): “While certain crossmodal correspondences may reflect the internalization of the statistics of the natural environment (and may perhaps best be modelled in terms of coupling priors according to Bayesian integration theory; Ernst, 2007), others may result from the peculiarities of the neural systems w”
Causal Inference in Multisensory Perception (2007, background): “Further evidence for this idea comes from a study that showed that by providing evidence that two signals are related it is possible to incite subjects to more strongly combine two cues [21].”
Multisensory Causal Inference in the Brain (2015, background): “This approach has provided invaluable insights about various aspects of perceptual multisensory integration [3,4,12,13] and helped to identify the sensory computations likely to be carried out by the underlying neural processes [14,15].”
On tasty colours and colourful tastes? Assessing, explaining, and utilizing crossmodal correspondences between colours and basic tastes (2015, positive): “By contrast, novel crossmodal correspondences can, in certain cases at least, be learnt over a surprisingly small number of exposures to a new statistical regularity in the environment or psychophysics laboratory [17].”
The temporal binding window for audiovisual speech: Children are like little adults (2016, background): “Modeling studies suggest that the final product of a multisensory interaction is based on the statistical weighting of the combined unisensory signals, with these weights being inversely proportional to stimulus reliability (Ernst, 2007; Helbig & Ernst, 2007; Witten & Knudsen, 2005).”