Credit: CC0 Public Domain Emilia Gómez, a researcher with the Musical Technology Research Group (MTG) of the Department of Information and Communication Technologies (DTIC) at UPF, has attempted to create new digital approaches to enrich the experience of a classical music concert before, during and after the concert itself, in order to bring classical music to new audiences in an innovative way via technology.
"One of the applications we investigated in the post-concert context was a type of facility in which the user takes on the role of conductor so that their movements 'conduct' a virtual orchestra. This type of facility has been successful in places such as the House of Music in Vienna," says Álvaro Sarasúa, first author of a study published on 25 February in the journal Frontiers in Digital Humanities, led by Emilia Gómez.
In human-computer interaction, interface metaphors provide users with a way to interact with the machine that resembles a known activity. These metaphors are widely used in digital musical instruments. In this study, the authors used the conductor metaphor. The interest in using these metaphors is so that the user, in a virtual facility in which to control the tempo and dynamics at which an orchestra plays, can apply their knowledge or their prior intuition about how a certain activity functions in the real world—in this case, conducting an orchestra.
"From the research point of view, in this study, we were interested in determining how intuitive notions available to users of this kind of facility may be explicitly used by the system to respond to their gestures of conducting an orchestra more intuitively, and therefore, may provide a more expressive performance," says Sarasúa.
In the initial stages of the project, the researchers conducted observational studies in which they asked people with different degrees of musical training to listen to several excerpts of classical music and then, while the music was playing again, to make body movements they would make while conducting an orchestra to communicate various aspects such as tempo and intensity. They recorded the participants' movements using a Kinect camera, which recorded the position of various parts of the body.
Different interpretative tendencies observed
Analysis of these data revealed different tendencies between subjects. On the one hand, to communicate the tempo of the music, there were people who consistently communicated beat somewhat ahead of the tempo of the music, while others did so in total synchrony or with some delay. Regarding the dynamics with which the instruments played, some people tended to move more energetically during the forte parts, while others simply raised their arms more.
"Our starting hypothesis for this study was that a system that allows controlling the tempo and volume of a piece of music by movements mimicking those of a conductor may be more intuitive if it can be adapted to these personal tendencies," explains Sarasúa
"In the example of our experiment, there is an 'implicit' advantage by using the metaphor, in that by telling the user that they must 'conduct' the orchestra, they know that they must move their arms and have some intuition as to their ability to control tempo and dynamics. Using the proposed system, this improvement is more explicit in that the system observes and analyses what the user senses they must do and adapts to perform according to these intuitions."
The authors refer to this strategy as mapping by observation, since mapping, or the way in which the user's movements are connected to the resulting music, specifically adapts for each user by observing what they do when asked to perform movements imitating a conductor without giving any specific instructions.
In evaluating the system, the researchers observed 24 participants who performed first with a baseline system that did not adapt to each user, and then a system based on the mapping strategy, and compared the results. This comparison was carried out in a context in which users learned to "conduct" the virtual orchestra through practice. The two systems were compared on the basis of objective and subjective measures on a series of tasks in which the participants had to make the orchestra play at different volumes and do so in synchrony with the metronome at different tempos.
The usability of the tailored system is better
The results of the experiment showed that the usability of the tailored system is better, both in terms of providing more intuitive control of the dynamics with which the user conducts the orchestra, and in terms of having more precise control over the tempo. Furthermore, the results also show a strong correlation between measures that can be drawn from the data used to make the adaptation and improvement introduced by the system, which indicates that it is possible to estimate in advance the extent to which the spontaneous movements of a particular user will be really useful to adapt the system to their style.
The findings of this study are particularly relevant in contexts where interaction is designed from using interface metaphors, as is the case of the conductor of an orchestra. "In the example of our experiment, there is an "implicit" advantage in using the metaphor, in that by telling the user that they must "conduct" the orchestra, they know that they must move their arms and have some intuition as to controlling tempo and dynamics. Using the proposed system, this improvement is more 'explicit' in that the system observes and analyses what the user senses they must do and adapts to perform according to these intuitions," the authors conclude.
Explore further: UK virtual orchestra puts you in conductor's stand
More information: Álvaro Sarasúa et al, Mapping by Observation: Building a User-Tailored Conducting System From Spontaneous Movements, Frontiers in Digital Humanities (2019). DOI: 10.3389/fdigh.2019.00003