What are the implications of being able to "listen to a choreography" or "feel a ballet?" Despite the importance of observing the bodily movements of others to understand and predict actions, little is known about the underlying plasticity of the neural mechanism.
The blending of the arts and sciences, traditionally viewed as being somewhat at cross-purposes, has found fertile ground in areas such as education, therapy and rehabilitation. The EU-funded DANCE project found this to be especially true when combined with human-centred computing innovations.
The DANCE project explored how bodily movement can be expressed and perceived in auditory terms. The project developed technologies which measured the expressive qualities of full-body movement in real-time, translating gestures from visual to auditory phenomena. While contributing to brain-imaging which maps the visual and auditory cortex of blind and sighted people, the approach also offers therapeutic potential, as well as novel human-machine non-verbal interfaces.
Seeing through listening
Research has shown that the sonification (using non-speech audio to convey information) of movement can increase body awareness, along with heightened sensitivity to emotions. This finding has proven useful when applied to therapeutic endeavours.
DANCE brought together performance artists, sound designers, vision and social neuroscientists, computer scientists and engineers to further explore the potential for rehabilitation and increased social inclusion emerging from the sharing of spaces and emotions between visually impaired and sighted people. The EU-funded DANCE team have previously adopted the approach to help patients with Parkinson's disease and those suffering from chronic pain.
In the project's work, choreographed expressive movement actually creates sonic content in real time. As project coordinator Dr. Antonio Camurri, explains, "This means that dance itself is then conceived as a musical composition, changing its traditional dimension to offer a listening experience rather than principally a visual one."
An overarching theme of the project was to explore the way in which multisensory information runs the risks of creating sensory overload in perceivers. "So instead of augmenting the senses, with the consequent risk of blunting the experience, DANCE proceeded by deliberately creating sensory deprivation as a means of sharpening the senses." Dr. Camurri elaborates. In doing so DANCE also contributed to neuroscience studies looking at brain plasticity.
DANCE conducted a human functional magnetic resonance imaging (fMRI) study investigating the relationship between movement features, obtained from automated computational analyses of video clips, and the brain activity during perception of the videos.
The results showed that features requiring low-level computation, such as movement speed, map to specific brain areas related to early visual and motion sensitive regions. While mid-level features, such as expressive fluidity, are related to dynamic aspects of posture encoded in different brain areas.
As Dr. Camurri summarises, "Our computational feature based analysis suggests that the neural mechanism of movement encoding uses features that have a different neural basis from semantic categories. That is, movement perception is organized in the brain not so much by semantic categories, than by features of the body movements themselves."
Choreographing the next steps
The team developed a technology platform alongside freely available software libraries. These have been adopted for several public events, scientific experiments and mobile applications. A publicly available movement dataset was also created along with prototypes of applications for both blind and non-blind people.
The project's main public event entitled, "Atlante del gesto Genova' included the participation of blind and sighted people, with an institute for blind people, which took place from November 2016 to March 2017 and has created an on-going community of over 150 participants (including non-specialist members of the public).
The work can be adapted to generate creative content for therapy and rehabilitation. For example, a new joint laboratory with the Gaslini children's hospital in Italy was created during DANCE. Here, physiotherapists alongside computer engineers and researchers, collaborated in the creation of gaming for rehabilitation.
DANCE has also been a rich source of inspiration for further research directions such as neuroscientific studies seeking to reveal the underlying neural mechanismsinvolved in the perception of high-level, affective communication through movement, with the aim of creating generation of novel interfaces and applications.
No comments:
Post a Comment