Research essay on sound and image relationships: Understanding Creativity with Sound through Embodied Cognition

Also available on Interalia.

I'm interested in cross-modal cognition and how we structure creative processes through embodied experience. I’m particularly interested in creativity with sound. 


A central theme of my work is representing sound with image and image with sound to evoke, highlight and explore audio-visual relationships through texture, movement, light and colour; typically, I use a mixture of moving image, animation, illustration, photography and sound design. I use found objects and found sounds as materials in my work. To treat audio and visual elements without preference is important to me, so my workflow entails alternating image and sound production within a project to help create a natural reciprocation.


I’m currently researching how these kinds of cross-modal representations are possible from a cognitive point of view. To understand this, I’m looking at cognitive science and related fields such as linguistics and neuroscience. The language we use to describe sound suggests sound conceptualisation is metaphorical in nature. I argue that sounds are fundamentally understood in terms of objects, this can be expressed as the conceptual metaphor Sounds are Objects. Sounds can exist within space: 'up', 'down'; they can have texture: 'rough', 'soft'; they can have force: ‘piercing’, ‘powerful’; etc. These metaphors are reflective of the conceptual system we use to understand, communicate, and be creative with sound and importantly they have experiential, embodied bases. 

The fact that certain images and sounds can be representative of one another due to perceived similar characteristics, despite pertaining to totally separate physical phenomena and stimulating separate sense organs is particularly fascinating, and it allows for some interesting questions and audio-visual experiments. What’s the effect of juxtaposing visual sound representations with our ingrained conceptualisations of sounds? What kind of cognitive dissonance does this create - an 'upwards' sound with a downwards movement, for example? Does showing a large object with a ‘small’ sound project largeness to the sound? Does it project ‘smallness’ to the object? Does it do nothing at all? In what way can the perception of sound and image be altered through these contrasts?


MA Art and Science

2015 - 2017 Central Saint Martins

BA (hons) Digital Music and Sound Arts

2010 - 2013 University of Brighton