The Journal of Sonic Studies
Journal of Sonic Studies, volume 6, nr. 1 (January 2014)Daniel Hug; Moritz Kemper: FROM FOLEY TO FUNCTION: A PEDAGOGICAL APPROACH TO SOUND DESIGN FOR NOVEL INTERACTIONS

To refer to this article use this url: http://journal.sonicstudies.org/vol06/nr01/a03

The Resulting pedagogical Framework

Required Preliminary Competencies

next section

For the overall research strategy to yield the desired results it has to be ascertained that all participants share a comparable and sufficient level of sound-related competence. While not required for the initial stage, later development stages require some basic knowledge of sound editing and multitrack composition with simple effects. Here we focus on a range of methods that originate in tape-based sound editing and basic signal processing methods, which can be understood and reproduced with relative ease. And, as we are also dealing with functional prototypes, another precondition is a basic knowledge of electronics and programming.

Stages of the Process

For developing the sound concepts, we used a range of prototyping methods, which we arranged in a three-stage process. They fulfill the relevant pedagogical requirements described above:

  • They allow us to deal with performativity and are open for improvisation and dialogical exploration, by employing live, ad-hoc sound making techniques
  • They rely on tools which are accessible, easy to learn, but offer a potential for complexity and iterative refinement
  • The sound making techniques and tools are integrated in such a way that continuous development from open exploration to functional implementation is possible

For the exploration of the “possible futures” through design, and to make sure that the resulting artifacts also can serve as cases for further study, we need to produce cases of “plausible experiences“ that can be analyzed in the group. This is a common practice in design called “experience prototyping”, which is described as “any kind of representation, in any medium, that is designed to understand, explore or communicate what it might be like to engage with the product, space or system we are designing” (Buchenau and Suri 2000: 425). In our case, we used a variation of the Wizard-of-Oz prototyping method,[4] where a computational system is simulated by an invisible human, triggering events in real time, while another person acts out the scenario or uses the prototype. This satisfies the requirement of providing performative, improvisational real time prototypes and provides the basis for the final stage, the implementation of a functional prototype.

In order to encourage the dialogical process, during all demonstrations and at the end of each stage, the experience is discussed with the whole class. The demonstrations and the discussions are also recorded on video, which is not only important for the ongoing reflection, but also for the underlying research effort described above.

In the following we will describe the resulting stages in more detail.

Stage 1: Foley Mockup

In this stage, we focus on exploring the richness of sound as material and make it possible to develop an initial concept that can be implemented in the form of a live demonstration of a Wizard-of-Oz mockup.

As mentioned above, our methods need to be accessible to someone with very little previous experience and open for heterogeneous groups with different backgrounds in terms of sound and music production. Foley, the technique of creating sound effects synchronous to film, is ideal for this and can be easily learned through a set of simple exercises (e.g. “Semantic Foley”, see [Hug 2010b]). Combined with the possibility of making sounds with the voice and the body, it offers a vast amount of sonic possibilities with a very simple technical setup (Ament 2009). One technical element, however, is essential: Just as in actual Foleying, the sounds made must be captured with a microphone. The microphone is the first element of an electroacoustic sonic transformation and provides an aesthetic link to the following step in the design process, the electroacoustic mockup. The microphone, and the projection of sound through a loudspeaker, provide an acousmatic listening condition, which further supports the understanding of sound transformation in the electroacoustic condition.

As the temporal and technical investment in creating the sounds for the Foley stage is relatively low, another requirement is fulfilled, which is the openness to change and the creation of design alternatives. Also, it makes it easier to abandon design directions, or to “kill your darlings”[5] if necessary.

Finally, this method is also particularly open to improvisation that supports a dialogical form of design, where a sonic idea can be immediately adapted, based on the reactions of users and the audience, which, being unexpected, may often challenge interaction assumptions.


Figure 1: Performing the Foley mockup

In the workshop, after some introductory exercises, the actual development of the Foley mockup commences right from the beginning. The students go through the usual steps of research and concept development and are asked to develop an application scenario, which needs to be plausible within possible everyday settings. Also, the narrative and performative quality of the interaction element and how it could be manifested in sound should be considered. The Foley mockup is then performed, using the Wizard-of-Oz method. Ideally, somebody from the audience performs the interaction. Afterwards, the demonstration is discussed with all participants, and ad-hoc suggestions might be tried out immediately, using the Foley technique. Thus, the Foley mockup stage serves as both proof of concept and playful inspirational session.

Stage 2: Electroacoustic Mockup

Building on the results of the Foley mockup, the next step uses recorded sounds that are played back via a multisampler setup. In didactical terms, and also in terms of design iteration, this stage serves to connect the Foley stage with the final stage, where the interactive systems are implemented as functional prototypes.

The use of multitrack editing allows students to create complex sounds in a more controlled way than is possible with Foley. Still, the experience and sound ideas from the Foley stage can be used as a starting point. The resulting sounds can then be triggered with the multisampler software, and real time mappings can be explored and performed by the “wizard” with a MIDI keyboard. These tools require a more structured approach already, in terms of defining mappings and producing assets, but are still easy to understand and handle and relatively flexible to modify.

The overall process of this stage is the same as in the Foley stage: The participants are asked to refine the initial ideas into a specific application scenario, considering the design requirements of the situational context, and to work towards a systematic approach to defining sound assets, modification parameters, and mappings to specific interaction elements. They are asked to perform their interaction scenario, using MIDI controllers to explore the dynamic relationships of sound properties, action and artifact.

Finally, again, the interactive scenarios are performed in a Wizard-of-Oz setting, and participants can try the interactions themselves. Also at this stage, the participants are asked to consider designing for performative and expressive variation and adaptation and to stay open to improvisation.


Figure 2: Two wizards using a MIDI keyboard to perform the electroacoustic mockup

Stage 3: Functional Prototype

The exploration of the context and the possibilities of sounding interactive artifacts, which have been developed in the previous stages, lead us to the third stage, where the students are asked to develop a functional prototype of the envisioned system. This stage is crucial because it requires, on one hand, a clear vision of the concept and offers, on the other hand, the possibility for a broader audience to interact with the designed artifact or system. The electroacoustic mockup forms an ideal basis for the functional prototype, as the students are able to use their digital sounds implemented in the multiplayer environment and simply replace the “wizard” through a combination of sensor technology and programming.

The key challenge in this stage is to be able to abstract from real world situations to single parameters and eventually come to simple yet effective solutions that also take into account the chosen context. In our case of interactive sound design, this means using appropriate interfaces between input data and the sound production environment. Therefore, students are using established prototyping tools (e.g. Processing, Arduino) together with improvised combinations of different sensor platforms (Microsoft Kinect, distance sensors, microphones).


Figure 3: Working on the electronics for the functional prototype

The students benefit from the previous stages in the process: Both the situation and the direction of the sound design have been defined. Likewise, the students already have developed the basic mapping intuitively by exploring parameter controls with the MIDI keyboards. From this basis, an initial decision about the technology to be used can be made. Guidance in the early stages is useful in order to train focus on the most relevant key events in the concept, which could be used within the setting of a functional prototype, as it is hard to fully implement a whole system in the short time of the workshop. This step also means scaling down the initial expectations from the participants regarding the possibilities of technology.