Multisensory integration, sensory recalibration, and the reduction of cross-modal perceptual discrepancies
Final Report Abstract
Locations of objects and bodily parts can be sensed by different modalities, but the respective perceptual estimates often differ from each other because of noisy sensory systems and variations of bodily and environmental states. We adjust to such discrepancies by means of multisensory integration and sensory recalibration (in motor tasks accompanied by motor adaptation). Multisensory integration is observed in judgments of discrepant bimodal stimuli, e.g. visual and auditory stimuli at different positions. Sensory recalibration is observed with unimodal stimuli that follow one or several discrepant bimodal stimuli. In both cases the perceived position of each stimulus is biased toward the position of the concurrent or preceding discrepant stimuli. This project served to identify commonalities and differences of integration and recalibration across two experimental paradigms, one focusing on audio-visual signals and one on visuo-motor signals. Multisensory integration and sensory recalibration are beneficial only when discrepant position estimates have a common cause. Their strength should be related to explicit common-cause judgments and common-cause evidence such as synchrony or correlated movements. Supporting our main hypothesis, we found that the strength of integration and recalibration depend on common-cause evidence on different time scales. Integration is related to common-cause judgments and evidence on a short time scale, whereas recalibration likely depends on evidence at a longer time scale. This difference is consistent with another key result obtained here: that integration is instantaneous, depending on the current spatial discrepancy and independent of preceding spatial discrepancies, whereas recalibration is cumulative. Under natural conditions there is more than one visual stimulus which could be related, e.g., to an auditory stimulus. Which one is selected for integration and/or recalibration? For both experimental paradigms our studies show that not a single visual stimulus is selected, but that the perceived position of the tone (or the hand) is attracted by the positions of two visual stimuli in a graded manner. The impact of each visual stimulus depends on the specific commoncause evidence, manipulated by different temporal offsets, but also by the configuration of all stimuli. Multisensory integration and recalibration are asymmetric. For example, the perceived position of an auditory stimulus is shifted strongly toward the position of a synchronous visual stimulus, but the perceived position of the visual stimulus is only hardly affected. Both for integration and recalibration the asymmetry can depend on the relative reliabilities of the discrepant estimates. We show that this dependency exists even when relative reliabilities are manipulated by means of the type of perceptual judgment required. Thus, the consequences of reliabilities are shaped not only by sensory processes, but also by the related actions.
Publications
-
Visuo-proprioceptive integration and recalibration with multiple visual stimuli. Scientific Reports, 11(1).
Debats, Nienke B.; Heuer, Herbert & Kayser, Christoph
-
Cumulative multisensory discrepancies shape the ventriloquism aftereffect but not the ventriloquism bias. Cold Spring Harbor Laboratory.
Kayser, Christoph; Park, Hame & Heuer, Herbert
-
Cumulative multisensory discrepancies shape the ventriloquism aftereffect but not the ventriloquism bias. PLOS ONE, 18(8), e0290461.
Kayser, Christoph; Park, Hame & Heuer, Herbert
-
Different time scales of common‐cause evidence shape multisensory integration, recalibration and motor adaptation. European Journal of Neuroscience, 58(5), 3253-3269.
Debats, Nienke B.; Heuer, Herbert & Kayser, Christoph
-
Short-term effects of visuomotor discrepancies on multisensory integration, proprioceptive recalibration, and motor adaptation. Journal of Neurophysiology, 129(2), 465-478.
Debats, Nienke B.; Heuer, Herbert & Kayser, Christoph
-
Both stimulus‐specific and configurational features of multiple visual stimuli shape the spatial ventriloquism effect. European Journal of Neuroscience, 59(7), 1770-1788.
Kayser, Christoph; Debats, Nienke & Heuer, Herbert
