Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions

Holle, Henning, Obleser, Jonas, Rüschemeyer, Shirley-Ann and Gunter, Thomas C. (2010) Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. NeuroImage, 49 (1). pp. 875-884. ISSN 10538119

[img]
Preview
PDF
Download (1MB) | Preview
[img]
Preview
PDF
Download (4MB) | Preview

Abstract

Iconic gestures are spontaneous hand movements that illustrate certain contents of speech and, as such, are an important part of face-to-face communication. This experiment targets the brain bases of how iconic gestures and speech are integrated during comprehension. Areas of integration were identified on the basis of two classic properties of multimodal integration, bimodal enhancement and inverse effectiveness (i.e., greater enhancement for unimodally least effective stimuli). Participants underwent fMRI while being presented with videos of gesture-supported sentences as well as their unimodal components, which allowed us to identify areas showing bimodal enhancement. Additionally, we manipulated the signal-to-noise ratio of speech (either moderate or good) to probe for integration areas exhibiting the inverse effectiveness property. Bimodal enhancement was found at the posterior end of the superior temporal sulcus and adjacent superior temporal gyrus (pSTS/STG) in both hemispheres, indicating that the integration of iconic gestures and speech takes place in these areas. Furthermore, we found that the left pSTS/STG specifically showed a pattern of inverse effectiveness, i.e., the neural enhancement for bimodal stimulation was greater under adverse listening conditions. This indicates that activity in this area is boosted when an iconic gesture accompanies an utterance that is otherwise difficult to comprehend. The neural response paralleled the behavioral data observed. The present data extends results from previous gesture-speech integration studies in showing that pSTS/STG plays a key role in the facilitation of speech comprehension through simultaneous gestural input.

Item Type: Article
Keywords: Language, fMRI, multisensory, audiovisual, inverse effectiveness
Schools and Departments: School of Psychology > Psychology
Subjects: Q Science > QZ Psychology
Depositing User: Henning Holle
Date Deposited: 15 Jun 2010
Last Modified: 11 Sep 2017 08:04
URI: http://sro.sussex.ac.uk/id/eprint/2376
Google Scholar:5 Citations

View download statistics for this item

📧 Request an update