Sample-level sound synthesis with recurrent neural networks and conceptors

Kiefer, Chris (2019) Sample-level sound synthesis with recurrent neural networks and conceptors. PeerJ Computer Science, 5. e205. ISSN 2376-5992

[img] PDF - Published Version
Available under License Creative Commons Attribution.

Download (852kB)

Abstract

Conceptors are a recent development in the field of reservoir computing; they can be used to influence the dynamics of recurrent neural networks (RNNs), enabling generation of arbitrary patterns based on training data. Conceptors allow interpolation and extrapolation between patterns, and also provide a system of boolean logic for combining patterns together. Generation and manipulation of arbitrary patterns using conceptors has significant potential as a sound synthesis method for applications in computer music but has yet to be explored. Conceptors are untested with the generation of multi-timbre audio patterns, and little testing has been done on scalability to longer patterns required for audio. A novel method of sound synthesis based on conceptors is introduced. Conceptular Synthesis is based on granular synthesis; sets of conceptors are trained to recall varying patterns from a single RNN, then a runtime mechanism switches between them, generating short patterns which are recombined into a longer sound. The quality of sound resynthesis using this technique is experimentally evaluated. Conceptor models are shown to resynthesise audio with a comparable quality to a close equivalent technique using echo state networks with stored patterns and output feedback. Conceptor models are also shown to excel in their malleability and potential for creative sound manipulation, in comparison to echo state network models which tend to fail when the same manipulations are applied. Examples are given demonstrating creative sonic possibilities, by exploiting conceptor pattern morphing, boolean conceptor logic and manipulation of RNN dynamics. Limitations of conceptor models are revealed with regards to reproduction quality, and pragmatic limitations are also shown, where rises in computation and memory requirements preclude the use of these models for training with longer sound samples. The techniques presented here represent an initial exploration of the sound synthesis potential of conceptors, demonstrating possible creative applications in sound design; future possibilities and research questions are outlined.

Item Type: Article
Additional Information: Dataset for this research can be found at related URL.
Keywords: Music, Sound Synthesis, Machine Learning, Recurrent Neural Network, Conceptor
Schools and Departments: School of Media, Film and Music > Music
Research Centres and Groups: Centre for Research in Creative and Performing Arts
Subjects: M Music. Literature on music. Musical instruction and study > M Music
Q Science > Q Science (General) > Q0300 Cybernetics
Q Science > QA Mathematics > QA0075 Electronic computers. Computer science
Related URLs:
Depositing User: Chris Kiefer
Date Deposited: 17 Jun 2019 11:26
Last Modified: 10 Jul 2019 16:24
URI: http://sro.sussex.ac.uk/id/eprint/84328

View download statistics for this item

📧 Request an update