The Journal of Sonic Studies
Journal of Sonic Studies, volume 6, nr. 1 (January 2014)Daniel Hug; Moritz Kemper: FROM FOLEY TO FUNCTION: A PEDAGOGICAL APPROACH TO SOUND DESIGN FOR NOVEL INTERACTIONS

To refer to this article use this url: http://journal.sonicstudies.org/vol06/nr01/a03

Case Studies

In the following, we will present and discuss three cases that emerged during a workshop held at Zurich University of the Arts in November 2012. The workshop lasted twelve days in three consecutive weeks. Fourteen bachelor students participated, eleven of whom were male. None of them had a previous background in sound design, except for the courses providing the fundamental knowledge described above.

In order to provide a suitable starting point and a fast progression from the course start to production, four initial scenarios were given to the students, and groups were formed around them. These scenarios also functioned as design constraints, helping to keep the focus on a specific application context, motivated by core characteristics of use, and the overall situation. To assure consistency with previous research, they were based on the conceptual framework for interactive commodities described in (Hug 2010a) and (Hug 2013).

The following assignments/design scenarios were given:

Doc-o-Mat: A wearable or implant with a body-related purpose (“quantified self”): e.g. health monitor for specific genetic dispositions, nutritional values, fitness. It is designed to work in private and public settings.

Matchmaker: A wearable or implant that manages social relationships in public environments or works in relation to the experience of social situations. It may offer means for wearers to express themselves and communicate in a social setting. It may work at various levels of intimacy.

Toyz: This scenario is less defined by a specific situational context, but focuses instead on interactions that afford playful behavior, or that specifically address children.

Project 1: “Dipalu”

next section

Maria Antonieta Diaz, Patrick Müller, Marc Schneider

This project was based on the assignment of creating a toy. The initial idea of the group was to create something that could be used in the kindergarten. There was a strong relation to a real-world application, because one participant’s sister worked in childcare. Two directions were followed in the beginning: One was a device that would help to collect and share sounds; the other proposed a pet-like artifact that could mediate interactions between children and teacher.


Figure 4: Using simple materials to present the Foley mockup

Foley Mockup

In the Foley mockup, the group acted out both of their scenarios. The implementation of the Foley mockup was very simple. The participants used only their voice, both for mimicking the sound collector and for the character’s voice. In order to represent the pet-like artifact, a simple toilet roll was used. This approach allowed them to prepare the prototype with little effort. On the other hand, the sonic elaboration and variability in interaction was rather limited, and there were some insecurities during the demonstration due to a lack of practice, as they spent quite a lot of time sketching storyboards and discussing ideas.

During the demonstration, it became noticeable that the group enjoyed their idea, and producing sounds in real time in front of an audience was no trouble. Several sonic ideas were based on ad-hoc improvisation. But the actor needed to be in eye contact with the wizard almost constantly, which disrupted the presentation somewhat. Also, he behaved like an actor on a stage rather than a casual “user”. Another issue was the audibility of the sounds, which lead to follow-up explanations about design aspects.

Electroacoustic Mockup

For the development of the electroacoustic mockup, a refined scenario was developed, focusing on the idea of a pet-like character that could be placed inside a kindergarten. It was meant to be able to react to sonic situations as well as influence the children’s behavior by various vocal utterances, depending on the noise level. In this way nursery teachers could use it as a mediator for getting children to be quiet and listen or to stimulate their interest in a specific situation.

The group discussed situations that could occur when a large number of children are in one room and created visualizations of these situations. This helped the group to define key interaction states between the object and the children (e.g. children being noisy, quiet, talking, singing, etc.). By doing so, they were then able to systematize the expressive reactions of their object and the accompanying sounding qualities. Starting with simple sounds, the group expedited the sound through several iterations. Building on the Foley mockup, they worked with their voices, developing a systematic catalogue of vocal utterances. They edited the voices to make them more abstract and explored various expressive real time controls.

The demonstration performance still contained some improvisation, but when the audience challenged the interaction, e.g. by reacting differently than expected, the prototype turned out not to be so flexible anymore.

Functional Prototype

In the final step, the group found not only a way to integrate the functions of their toy into a single object, but they even designed an entire narrative around the object. They created a furry creature, gave it the name “Dipalu”, and created an illustrated background story for it.

The tracking of the loudness was realized with a simple microphone, attached to a notebook computer. They developed a program that scanned for sonic patterns based on noise level (e.g. children talking or screaming). These situations were then mapped onto MIDI notes that were sent to the multisampler software where the predefined sonic feedback was triggered.


Figure 5: The final object and hand-drawn storyboard from the Dipalu group

With this setup, the group managed to create a tangible experience of the product and situation. The final result was still very close to the initial approach and demonstrates how the method proposed here can work ideally: exploring interactive sound through simple tools at the beginning of the process and finalizing the concept by building a working prototype at the end.

Concluding Remarks

As mentioned, this group managed to closely follow the proposed method. At the beginning, their Foley mockup helped them in improvising and exploring several ideas. The live performance of the electroacoustic mockup lead to a more systematic yet still flexible approach. The group reports in their project diary:

“Now we were no longer experimenting, and we had to record certain emotions for this conceptual creature to come alive. We made a basic interaction board.”

This statement shows on the one hand the successful transition to a more systematic approach, but also that experimentation - in particular with interactive processes - was abandoned a bit early.

Despite the simple drafting methods, the actual development of sounds was a core challenge to many students. The “Dipalu” group reported:

“It was still a difficult process because we didn’t have a clear picture in our heads of what this creature was; we only had a sketch, around which we have constructed our story.”

It turned out that this storytelling was an important catalyst for making sound design decisions.

Furthermore, this group demonstrated that the methods we are describing are suitable and even enjoyable for non-sound designers for developing sounds in a heterogeneous team. They produced their sounds, “everyone contributing with unique sounds; it was a long but fun process.” The quality of the sounds produced was surprisingly high, considering their limited experience. During the different stages they discovered the value of creating rich, varied sonic material and also took the opportunity to explore sounds outside of the narrow confines of the interaction script or the requirement specifications.

Project 2: Wash and Play

Basil Schmid, Nils Solanki, Ramun Rinklin

For this group, the starting assignment was the “Doc-O-Mat”. They soon focused on the process of hand washing, which has become more widely discussed again in the context of swine flu. The group’s aim was to make the process of washing the hands according to the WHO guidelines more enjoyable through “gamification”.[6]

Foley Mockup

The concept of gamification lead the group to the conclusion that “game-like sounds” in the form of synthetic 8-bit sounds as we know from old video game consoles should be used, with the aim of transferring an aesthetic of their gaming experiences into an everyday context (washing hands). Another argumentation was that synthetic beeps were already part of the soundscape in a hospital, and thus their design approach would work because the basic sonic aesthetics were familiar and still offered a novel, playful quality to the chore of washing one’s hands. While these are valid positions, the result was that the “Foley stage” in this project was actually inexistent.

Electroacoustic Mockup

Because the first prototype turned out to be quite linear, with game sounds being triggered at each step of the process of washing hands, we encouraged the team to further explore the motivational and gestural aspects. In particular they were asked to investigate potentials for interactive sound-action relationships, to help increase the efficiency of washing hands, which, according to the WHO guidelines, depends on hand positions and the correct execution of specific motions.

The group thus investigated the habits of people when washing their hands and created a step-by-step overview of the process. They decided to work with spatial position, the overall temporal development, and timing and strength of hand movement. Still, the design remained rather linear. Further qualitative research would have been necessary in order to find interesting action-sound couplings.

Sonically, the decision to recreate the 8-bit video game sound aesthetics was paramount. Being asked to explore variations and interactive relationships, they resorted to a software which generates 8-bit game style sounds based on the setting of some parameters. With this software it was possible for them to create a set of sound variations without departing from the aesthetic framework. On the other hand, the opportunity for a more open sonic exploration was missing. This was also a consequence of their interaction concept, which was focused on signal sounds for each consecutive stage of hand washing, and did not consider more expressive and interactive potentials, or how sound could be used to motivate people to wash their hands more thoroughly.

Functional Prototype

The team of “Wash and Play” was very motivated to implement their prototype in the final phase, and they stated in a discussion that for them the whole project depended on successful functional implementation. During this stage, the group was still limited in terms of sound design by their chosen style. Also, their goal of implementing the prototype into an actual restroom lead to long experimentation and development with the sensors and electronics, which had to be fitted into the faucet and soap dispenser.


Figure 6: Sensor being used by the Wash and Play group to track running water

On the one hand, the team profited from this effort, because it was very rewarding to experience the interaction in a realistic setting. Also, the challenge led to some innovative technical solutions from which the group learned a lot.

VideoObject 1: Wash and Play final presentation

In terms of the workshop goals, the consequence of this technical absorption was that even the few possible approaches to investigate and develop truly interactive aspects and the corresponding sounds, for instance during the rubbing of the hands, as well as a more thorough investigation of the topic of hygiene, could not be further explored.

Concluding Remarks

Although the 8-bit game aesthetics laid on its usual charm, the style-based approach chosen by this group did not correspond well with the proposed methodology; maintaining the style restricts the creation of sounds and the sonic flexibility required for modulations in line with interactions. Also, there is the danger of producing sonic stereotypes that can soon become tiresome. The discussions with the group revealed that there was a certain degree of distrust that any sound could possibly be interesting enough, which resulted in an orientation towards sounds that were considered “cool” – in this case a certain type of game sound. The argument that synthetic beeps are already common in hospital and health center contexts seemed valid, but also leads to the reinforcement of a (problematic) sonic condition rather than offering alternative approaches.

Last but not least, the absence of an actual Foley-mockup precluded innovative insights from an improvisational setting and the mental readiness to abandon some design ideas if necessary. One related issue was the focus on using sounds to notify and represent certain stages rather than to explore expressive, performative, and narrative qualities.

Project 3: “Sonotag”

Daniel Mischler, Matthias Kappeler, Meret Vollenweider, Oliver Kalbermatter

This project emerged from the assignment “Matchmaker”. The group’s initial intention was to focus on the slightly awkward social situation that comes into being when two or more people are in an elevator and the resulting issues of interpersonal communication.

In their initial research the group investigated typical elevator scenes from movies in order to better understand the narrative dimensions associated with the setting. The existing use of sound in the form of “elevator music” was judged to be insufficient or even counterproductive. Thus, the approach of the group finally was directed at stimulating movement through sound, in order to relieve the awkwardness of standing in a certain position.


Figure 7: Acting out the Foley mockup of the Sonotag group

Foley Mockup

The Foley mockup employed two pitched tones, representing two people standing in an imaginary elevator. The pitch of the tones was mapped to the distance between them. The goal was to make the two pitches match by moving around in the elevator.

The intention of the group was to confuse both participants by using strange and alien tones, requiring them to cooperate in resolving the confusion and thus reducing the inhibition to interact with each other. The sounds were produced by voice. This had the advantage that the resulting sounds had a complex texture and were not too static, despite being pitched. The disadvantage was that there were interruptions of the sustained tone caused by breathing, which could be misunderstood as the beginning of a new sound event. Also, the voice was not strange or alien enough for the desired effect of confusion, as it was recognizable as being the voice of a human being. Moreover, this recognizability made the sound rather funny.

The mockup managed to clearly demonstrate the principle of interaction. Nevertheless, the group could not explore the full potential of sound as a socializing element, their focus on representation of distance alone being too technical and obvious.

Electroacoustic Mockup

For the electroacoustic mockup, the group used a heavily pitched version of the recorded voice that could be looped continuously. This provided two advantages: First, the voice was defamiliarized and indeed turned into a strange, alien sound that had an identity of its own, without revealing the human origin. Second, it could be played continuously and more precisely, as the setup offered better control of duration and pitch.

During the demonstration they experienced difficulties finding the right match between action and the triggering of the tones. Also, while this stage was very similar to the Foley mockup, it actually reduced the complexity of sonic expressions and interactions.

Functional Prototype

As the previous stage resulted in even more limitations for interaction, the group decided to turn the basic principle of two people interacting through distance into a narrative game, which was named “Sonotag”. The two players took the roles of a rabbit and a wolf who meet each other in the middle of an imaginary playfield. They are blindfolded, and the only information from the game is provided through sounds delivered through a pair of headphones, which are wirelessly connected to a computer. This computer calculates the distances between the two protagonists as well as their distance to the border of the playfield. Based on these distances, different sonic feedbacks are played back to the participants: Both protagonists hear the distance of the opponent as a pitched voice sample, which is complemented with breathing and growling sounds when they are very near or being eaten as crunchy bite, and touching the border is audible as electric discharge. This latter sound event turned out to be very prominent and almost dominated the sonic experience. The reason for this was that the tracking system would break down if a participant went outside a certain area. Thus, a technical limitation forced the group to resort to a rather stereotypical warning sound, which did not fit into the overall narrative.

VideoObject 2: Sonotag final presentation

The functional mockup was quite consistent with the initial approach but abandoned the original situational context in favor of a dramatic game. Also, the group still used vocal sounds but added some illustrative or metaphorical sounds. The advanced tracking technology, based on a Microsoft Kinect camera, together with the tight restrictions imposed by the gameplay as well as issues associated with the height of the room, lead to a somewhat more challenging programming task. Furthermore, they needed to implement a wireless sound transmission solution, and the narrative nature of the game required a sound design that was rather cinematic. By splitting the group and distributing the work into the three domains (tracking of the players, the gameplay, and the sound design), they were able to meet these challenges. But an additional effort was needed to ensure aesthetic coherence.


Figure 8: Camera-based system for tracking the movement of the players for the functional prototype of Sonotag

Concluding Remarks

The initial idea for creating disturbing and alien sounds to break social barriers turned into a simple representation of the distance between people. This example demonstrates that if the mockup is too reductionist and simplified, an otherwise good concept is endangered, leading to aesthetic dead ends. Also, in this case, resorting to a narrative approach helped overcome this issue. The narrative, together with the gameplay requirements, gave clear shaping to the possible interactions and bridged the gap to the sound making process by defining a common aesthetics. In the end, however, the technology left its mark by forcing the group to implement a warning sound that they found inappropriate to the chosen sound aesthetic.