How gesture and speech interact during production and comprehension

[摘要] This thesis investigates the mechanisms that underlie the interaction of gesture and speech during the production and comprehension of language on a temporal and semantic level. The results from the two gesture-speech production experiments provide unambiguous evidence that gestural content is shaped online by the ways in which speakers package information into planning units in speech rather than being influenced by how events are lexicalised. In terms of gesture-speech synchronisation, a meta-analysis of these experiments showed that lexical items which are semantically related to the gesture's content (i.e., semantic affiliates) compete for synchronisation when these affiliates are separated within a sentence. This competition leads to large proportions of gestures not synchronising with any semantic affiliate. These findings demonstrate that gesture onset can be attracted by lexical items that do not co-occur with the gesture. The thesis then tested how listeners process gestures when synchrony is lost and whether preceding discourse related to a gesture's meaning impacts gesture interpretation and processing. Behavioural and ERP results show that gesture interpretation and processing is discourse dependent. Moreover, the ERP experiment demonstrates that when synchronisation between gesture and semantic affiliate is not present the underlying integration processes are different from synchronous gesture-speech combinations.

[发布机构] University:University of Birmingham;Department:School of English, Drama and American & Canadian Studies, Department of English Language and Linguistics

[关键词] P Language and Literature;P Philology. Linguistics