University of Sussex
Browse
1/1
2 files

Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions

journal contribution
posted on 2023-06-07, 15:25 authored by Henning Holle, Jonas Obleser, Shirley-Ann Rüschemeyer, Thomas C Gunter
Iconic gestures are spontaneous hand movements that illustrate certain contents of speech and, as such, are an important part of face-to-face communication. This experiment targets the brain bases of how iconic gestures and speech are integrated during comprehension. Areas of integration were identified on the basis of two classic properties of multimodal integration, bimodal enhancement and inverse effectiveness (i.e., greater enhancement for unimodally least effective stimuli). Participants underwent fMRI while being presented with videos of gesture-supported sentences as well as their unimodal components, which allowed us to identify areas showing bimodal enhancement. Additionally, we manipulated the signal-to-noise ratio of speech (either moderate or good) to probe for integration areas exhibiting the inverse effectiveness property. Bimodal enhancement was found at the posterior end of the superior temporal sulcus and adjacent superior temporal gyrus (pSTS/STG) in both hemispheres, indicating that the integration of iconic gestures and speech takes place in these areas. Furthermore, we found that the left pSTS/STG specifically showed a pattern of inverse effectiveness, i.e., the neural enhancement for bimodal stimulation was greater under adverse listening conditions. This indicates that activity in this area is boosted when an iconic gesture accompanies an utterance that is otherwise difficult to comprehend. The neural response paralleled the behavioral data observed. The present data extends results from previous gesture-speech integration studies in showing that pSTS/STG plays a key role in the facilitation of speech comprehension through simultaneous gestural input.

History

Publication status

  • Published

File Version

  • Accepted version

Journal

NeuroImage

ISSN

1053-8119

Publisher

Elsevier

Issue

1

Volume

49

Page range

875-884

Department affiliated with

  • Psychology Publications

Full text available

  • Yes

Peer reviewed?

  • Yes

Legacy Posted Date

2010-06-15

First Open Access (FOA) Date

2018-01-23

First Compliant Deposit (FCD) Date

2018-01-23

Usage metrics

    University of Sussex (Publications)

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC