Cross-modal correspondences enhance performance on a colour-to-sound sensory substitution device

Hamilton-Fletcher, Giles, Wright, Thomas D and Ward, Jamie (2016) Cross-modal correspondences enhance performance on a colour-to-sound sensory substitution device. Multisensory Research, 29 (4-5). pp. 337-363. ISSN 2213-4794

[img] PDF - Published Version
Restricted to SRO admin only

Download (1MB)


Visual sensory substitution devices (SSDs) can represent visual characteristics through distinct patterns of sound, allowing a visually impaired user access to visual information. Previous SSDs have avoided colour and when they do encode colour, have assigned sounds to colour in a largely unprincipled way. This study introduces a new tablet-based SSD termed the ‘Creole’ (so called because it combines tactile scanning with image sonification) and a new algorithm for converting colour to sound that is based on established cross-modal correspondences (intuitive mappings between different sensory dimensions). To test the utility of correspondences, we examined the colour–sound associative memory and object recognition abilities of sighted users who had their device either coded in line with or opposite to sound–colour correspondences. Improved colour memory and reduced colour-errors were made by users who had the correspondence-based mappings. Interestingly, the colour–sound mappings that provided the highest improvements during the associative memory task also saw the greatest gains for recognising realistic objects that also featured these colours, indicating a transfer of abilities from memory to recognition. These users were also marginally better at matching sounds to images varying in luminance, even though luminance was coded identically across the different versions of the device. These findings are discussed with relevance for both colour and correspondences for sensory substitution use.

Item Type: Article
Keywords: Sensory substitution, vision, colour, sound, touch
Schools and Departments: School of Psychology > Psychology
Subjects: B Philosophy. Psychology. Religion > BF Psychology
Depositing User: Jamie Ward
Date Deposited: 04 Jul 2016 10:56
Last Modified: 01 Jul 2019 19:00

View download statistics for this item

📧 Request an update