The Journal of Sonic Studies
Journal of Sonic Studies, volume 6, nr. 1 (January 2014)Daniel Hug; Moritz Kemper: FROM FOLEY TO FUNCTION: A PEDAGOGICAL APPROACH TO SOUND DESIGN FOR NOVEL INTERACTIONS

To refer to this article use this url: http://journal.sonicstudies.org/vol06/nr01/a03

Project 3: “Sonotag”

Daniel Mischler, Matthias Kappeler, Meret Vollenweider, Oliver Kalbermatter

This project emerged from the assignment “Matchmaker”. The group’s initial intention was to focus on the slightly awkward social situation that comes into being when two or more people are in an elevator and the resulting issues of interpersonal communication.

In their initial research the group investigated typical elevator scenes from movies in order to better understand the narrative dimensions associated with the setting. The existing use of sound in the form of “elevator music” was judged to be insufficient or even counterproductive. Thus, the approach of the group finally was directed at stimulating movement through sound, in order to relieve the awkwardness of standing in a certain position.


Figure 7: Acting out the Foley mockup of the Sonotag group

Foley Mockup

next section

The Foley mockup employed two pitched tones, representing two people standing in an imaginary elevator. The pitch of the tones was mapped to the distance between them. The goal was to make the two pitches match by moving around in the elevator.

The intention of the group was to confuse both participants by using strange and alien tones, requiring them to cooperate in resolving the confusion and thus reducing the inhibition to interact with each other. The sounds were produced by voice. This had the advantage that the resulting sounds had a complex texture and were not too static, despite being pitched. The disadvantage was that there were interruptions of the sustained tone caused by breathing, which could be misunderstood as the beginning of a new sound event. Also, the voice was not strange or alien enough for the desired effect of confusion, as it was recognizable as being the voice of a human being. Moreover, this recognizability made the sound rather funny.

The mockup managed to clearly demonstrate the principle of interaction. Nevertheless, the group could not explore the full potential of sound as a socializing element, their focus on representation of distance alone being too technical and obvious.

Electroacoustic Mockup

For the electroacoustic mockup, the group used a heavily pitched version of the recorded voice that could be looped continuously. This provided two advantages: First, the voice was defamiliarized and indeed turned into a strange, alien sound that had an identity of its own, without revealing the human origin. Second, it could be played continuously and more precisely, as the setup offered better control of duration and pitch.

During the demonstration they experienced difficulties finding the right match between action and the triggering of the tones. Also, while this stage was very similar to the Foley mockup, it actually reduced the complexity of sonic expressions and interactions.

Functional Prototype

As the previous stage resulted in even more limitations for interaction, the group decided to turn the basic principle of two people interacting through distance into a narrative game, which was named “Sonotag”. The two players took the roles of a rabbit and a wolf who meet each other in the middle of an imaginary playfield. They are blindfolded, and the only information from the game is provided through sounds delivered through a pair of headphones, which are wirelessly connected to a computer. This computer calculates the distances between the two protagonists as well as their distance to the border of the playfield. Based on these distances, different sonic feedbacks are played back to the participants: Both protagonists hear the distance of the opponent as a pitched voice sample, which is complemented with breathing and growling sounds when they are very near or being eaten as crunchy bite, and touching the border is audible as electric discharge. This latter sound event turned out to be very prominent and almost dominated the sonic experience. The reason for this was that the tracking system would break down if a participant went outside a certain area. Thus, a technical limitation forced the group to resort to a rather stereotypical warning sound, which did not fit into the overall narrative.

VideoObject 2: Sonotag final presentation

The functional mockup was quite consistent with the initial approach but abandoned the original situational context in favor of a dramatic game. Also, the group still used vocal sounds but added some illustrative or metaphorical sounds. The advanced tracking technology, based on a Microsoft Kinect camera, together with the tight restrictions imposed by the gameplay as well as issues associated with the height of the room, lead to a somewhat more challenging programming task. Furthermore, they needed to implement a wireless sound transmission solution, and the narrative nature of the game required a sound design that was rather cinematic. By splitting the group and distributing the work into the three domains (tracking of the players, the gameplay, and the sound design), they were able to meet these challenges. But an additional effort was needed to ensure aesthetic coherence.


Figure 8: Camera-based system for tracking the movement of the players for the functional prototype of Sonotag

Concluding Remarks

The initial idea for creating disturbing and alien sounds to break social barriers turned into a simple representation of the distance between people. This example demonstrates that if the mockup is too reductionist and simplified, an otherwise good concept is endangered, leading to aesthetic dead ends. Also, in this case, resorting to a narrative approach helped overcome this issue. The narrative, together with the gameplay requirements, gave clear shaping to the possible interactions and bridged the gap to the sound making process by defining a common aesthetics. In the end, however, the technology left its mark by forcing the group to implement a warning sound that they found inappropriate to the chosen sound aesthetic.