A conceptual model of audio-visual object formation and how it effects listening in noisy situations.
See full open access article here: http://dx.doi.org/10.1016/j.tins.2015.12.007.
How are auditory and visual stimuli combined (or not combined) in the brain?
Visual stimuli can exert higher order influence on the way humans parse complicated auditory scenes and the neural computations that underlie it. Studies of selective listening with audio-visual stimuli typically use speech tasks, and studies that employ non-linguistic stimuli usually only use a single audio-visual stimulus, rather than competing ones. The importance of visual information in noisy environments is easily demonstrable, but the perceptual and neural mechanisms that drive it are not well understood. We have developed a paradigm that allows manipulation of audio-visual coherence in artificial, non-linguistic stimuli with naturalistic dynamics, allowing generalization of the principles governing multisensory integration often obscured by the linguistic redundancies of speech. Our experiments are aimed at addressing the gap in the literature concerned with competing, cross-modal objects.
We are using novel behavioral paradigms and functional neuroimaging to investigate how temporal coherence between auditory and visual stimuli affects the way people interpret the auditory scene. This work will yield important insights about the way the brain efficiently makes use of multiple sources of information to form a single percept. It also may lead to assistive listening devices that rely on creating a helpful visual stimulus.