Journal of Sonic Studies, volume 6, nr. 1 (January 2014)Daniel Hug; Moritz Kemper: FROM FOLEY TO FUNCTION: A PEDAGOGICAL APPROACH TO SOUND DESIGN FOR NOVEL INTERACTIONS
Project 1: “Dipalu”
Maria Antonieta Diaz, Patrick Müller, Marc Schneider
This project was based on the assignment of creating a toy. The initial idea of the group was to create something that could be used in the kindergarten. There was a strong relation to a real-world application, because one participant’s sister worked in childcare. Two directions were followed in the beginning: One was a device that would help to collect and share sounds; the other proposed a pet-like artifact that could mediate interactions between children and teacher.
Foley Mockupnext section
In the Foley mockup, the group acted out both of their scenarios. The implementation of the Foley mockup was very simple. The participants used only their voice, both for mimicking the sound collector and for the character’s voice. In order to represent the pet-like artifact, a simple toilet roll was used. This approach allowed them to prepare the prototype with little effort. On the other hand, the sonic elaboration and variability in interaction was rather limited, and there were some insecurities during the demonstration due to a lack of practice, as they spent quite a lot of time sketching storyboards and discussing ideas.
During the demonstration, it became noticeable that the group enjoyed their idea, and producing sounds in real time in front of an audience was no trouble. Several sonic ideas were based on ad-hoc improvisation. But the actor needed to be in eye contact with the wizard almost constantly, which disrupted the presentation somewhat. Also, he behaved like an actor on a stage rather than a casual “user”. Another issue was the audibility of the sounds, which lead to follow-up explanations about design aspects.
For the development of the electroacoustic mockup, a refined scenario was developed, focusing on the idea of a pet-like character that could be placed inside a kindergarten. It was meant to be able to react to sonic situations as well as influence the children’s behavior by various vocal utterances, depending on the noise level. In this way nursery teachers could use it as a mediator for getting children to be quiet and listen or to stimulate their interest in a specific situation.
The group discussed situations that could occur when a large number of children are in one room and created visualizations of these situations. This helped the group to define key interaction states between the object and the children (e.g. children being noisy, quiet, talking, singing, etc.). By doing so, they were then able to systematize the expressive reactions of their object and the accompanying sounding qualities. Starting with simple sounds, the group expedited the sound through several iterations. Building on the Foley mockup, they worked with their voices, developing a systematic catalogue of vocal utterances. They edited the voices to make them more abstract and explored various expressive real time controls.
The demonstration performance still contained some improvisation, but when the audience challenged the interaction, e.g. by reacting differently than expected, the prototype turned out not to be so flexible anymore.
In the final step, the group found not only a way to integrate the functions of their toy into a single object, but they even designed an entire narrative around the object. They created a furry creature, gave it the name “Dipalu”, and created an illustrated background story for it.
The tracking of the loudness was realized with a simple microphone, attached to a notebook computer. They developed a program that scanned for sonic patterns based on noise level (e.g. children talking or screaming). These situations were then mapped onto MIDI notes that were sent to the multisampler software where the predefined sonic feedback was triggered.
With this setup, the group managed to create a tangible experience of the product and situation. The final result was still very close to the initial approach and demonstrates how the method proposed here can work ideally: exploring interactive sound through simple tools at the beginning of the process and finalizing the concept by building a working prototype at the end.
As mentioned, this group managed to closely follow the proposed method. At the beginning, their Foley mockup helped them in improvising and exploring several ideas. The live performance of the electroacoustic mockup lead to a more systematic yet still flexible approach. The group reports in their project diary:
“Now we were no longer experimenting, and we had to record certain emotions for this conceptual creature to come alive. We made a basic interaction board.”
This statement shows on the one hand the successful transition to a more systematic approach, but also that experimentation - in particular with interactive processes - was abandoned a bit early.
Despite the simple drafting methods, the actual development of sounds was a core challenge to many students. The “Dipalu” group reported:
“It was still a difficult process because we didn’t have a clear picture in our heads of what this creature was; we only had a sketch, around which we have constructed our story.”
It turned out that this storytelling was an important catalyst for making sound design decisions.
Furthermore, this group demonstrated that the methods we are describing are suitable and even enjoyable for non-sound designers for developing sounds in a heterogeneous team. They produced their sounds, “everyone contributing with unique sounds; it was a long but fun process.” The quality of the sounds produced was surprisingly high, considering their limited experience. During the different stages they discovered the value of creating rich, varied sonic material and also took the opportunity to explore sounds outside of the narrow confines of the interaction script or the requirement specifications.