File(s) under permanent embargo
Cross-modal correspondences enhance performance on a colour-to-sound sensory substitution device
journal contribution
posted on 2023-06-09, 02:00 authored by Giles Hamilton-Fletcher, Thomas D Wright, Jamie WardJamie WardVisual sensory substitution devices (SSDs) can represent visual characteristics through distinct patterns of sound, allowing a visually impaired user access to visual information. Previous SSDs have avoided colour and when they do encode colour, have assigned sounds to colour in a largely unprincipled way. This study introduces a new tablet-based SSD termed the ‘Creole’ (so called because it combines tactile scanning with image sonification) and a new algorithm for converting colour to sound that is based on established cross-modal correspondences (intuitive mappings between different sensory dimensions). To test the utility of correspondences, we examined the colour–sound associative memory and object recognition abilities of sighted users who had their device either coded in line with or opposite to sound–colour correspondences. Improved colour memory and reduced colour-errors were made by users who had the correspondence-based mappings. Interestingly, the colour–sound mappings that provided the highest improvements during the associative memory task also saw the greatest gains for recognising realistic objects that also featured these colours, indicating a transfer of abilities from memory to recognition. These users were also marginally better at matching sounds to images varying in luminance, even though luminance was coded identically across the different versions of the device. These findings are discussed with relevance for both colour and correspondences for sensory substitution use.
History
Publication status
- Published
File Version
- Published version
Journal
Multisensory ResearchISSN
2213-4794Publisher
BrillExternal DOI
Issue
4-5Volume
29Page range
337-363Department affiliated with
- Psychology Publications
Full text available
- No
Peer reviewed?
- Yes
Legacy Posted Date
2016-07-04First Compliant Deposit (FCD) Date
2016-07-04Usage metrics
Categories
No categories selectedLicence
Exports
RefWorks
BibTeX
Ref. manager
Endnote
DataCite
NLM
DC