Resumen de L'interazione nelle tecnologie musicali di realtà mista: il prototipo e-mocumu como esempio multimodale con propositi terapeutici
The main goal of this thesis is to study the interaction between the subject and the object within MIR (Mixed Interactive Reality) technologies from a therapeutic perspective. To see how this interaction works, I first consider the general structures of DMI (Digital Musical Interface) technologies, then look at the recently-developed motion capture system (MOCAP), and end by proposing six technologies of relevance to this context. These interactive technologies are significant in the HCI (Human Computer Interface) scenario, from both a historical and an applicative point of view.
The first and second parts of this paper serve to introduce the theoretical aspects of the interaction. All the technologies described are grounded in the interaction between the action and the subject’s perception of it. In the third part of the thesis, I describe proprioception, shaped by receptors that are fundamental both to movement and to self-awareness. I thus investigate the perceptive dimension of the interaction, referring to some fundamental theories on the philosophy of the body. This is useful for the purpose of taking a holistic perspective, based on the consideration of the human being as the subject, and the technological environment as the object. I then first discuss Presence, a concept acquired by other authors that I revisit within the interactive environment, which becomes a second “presence” interacting with the user. Then I introduce Gibson’s theory of visual perception, lingering on the concept of Affordance, which refers to the characteristics of the environment offered to the subject. I continue with the notion of IS (Image Schemata) and Johnson’s metaphor, which is functional to the comprehension of the MIR environment. The Sensorimotor Contingency Theory (SCT) advanced by O’Regan and Nöe is also helpful for elucidating how sensorimotor contingencies are involved in the perceptive process, within an interactive system. Finally, the Philosophy of Empowerment, interwoven with the aforementioned concepts, includes the experience and the perception that occurs with full control over our actions during the interaction. Bearing this theoretical basis in mind, we can understand the cross-modal interaction that takes place in a holistic view of the interaction between subject and object, in more than one sensory modality at a time. So it seems that images and sounds take part in this sensory integration that the user can experience within a MIR.
Tesi dottorale To understand how this sensory integration happens, we should go back to basics: how we perceive colors, and the link that can exist between color and sound. This correlation already exists in nature under the name of chromesthesia (a form of synesthesia), which is produced by an interaction of two sensory modalities in the brain.
In the fourth part of this paper, I offer a few conceptual contributions based on theoretical notions, adapted to the field of HCI, following a therapeutic trajectory. This research conceives music therapy as one of the appropriate fields for the application of this technology, which already existed in prehistoric times and has been reconsidered since the 20th century. Today, its efficacy is being studied by the scientific community. Several studies have shown that mixed and virtual technology can be a useful tool for supporting therapeutic interventions, to improve motor abilities, emotional and expressive capacities, and communication, for instance. My contribution concerns the interaction between the subject and the Presence (in the sense of the interactive environment), thus considering the action of the subject as mediated by perception, as suggested by Embodied Cognition.
Within this holistic approach, Multiple Affordances (MA) which are the possible paths made available to the subject by the Presence and determined by the subject’s exploration of the environment. This experience stems from a mapping done a priori. Enclosed by the MA, we find the Sensorimotor Maps (SM) that derive from the exploratory and perceptual process. These maps involve the creation of sensorimotor paths linked to changes produced by the information arriving from the environment with the performer’s movements. The concept of Creative Empowerment (CE) therefore relies on the MS, as happens when the subject gains full control over the technology, and is therefore capable of creative expression, self-control and awareness.
To better elucidate the salient features of a MIR technology for the music therapy setting, I developed a prototype and conducted an experiment during a period of research in Brazil (Unicamp). A crucial section of this thesis focuses on this research effort, which involved studying the correspondence between the sounds and colors to be used in the audiovisual design of the e-mocomu prototype, by implementing the aforementioned contributions and concepts. Emocomu stands for e-motion, color and music, and is an interdisciplinary technology developed with the primary aim of enabling users to control sounds and colors by means of their movements in space. The conceptual contributions take shape in the experimental work conducted, which underscores the potential of a music-therapy-oriented MIR technology. The experiment examined how participants interacted with their bodily performance by administering questionnaires on valence and arousal before and after the trial, and observing differences in the participants’ CE. The relationship between perception and action emerges, based on the crossmodal interaction, through MA, SM, and CE. This relationship is made possible by the mapping, a crucial step in any MIR technology, and by the concept of Presence as an “agent” that interacts with the subject.
Though still a limited prototype, e-mocomu is a good tool for identifying possible therapeutic applications of RMI technologies, and for observing the link existing between technology and creativity. In the last part of the thesis I thus propose some potential applications of a MIR technology in the therapeutic setting.
With this research, I hope to have offered a different, HCI-driven slant on the field of music therapy, as well as some helpful insight on the therapeutic process, both for the therapist and for the client.
