Journal of Sonic Studies, volume 6, nr. 1 (January 2014)Daniel Hug; Moritz Kemper: FROM FOLEY TO FUNCTION: A PEDAGOGICAL APPROACH TO SOUND DESIGN FOR NOVEL INTERACTIONS
Discussion and Recommendations
In this section we will discuss the observations that we have made during the workshop and present the resulting recommendations in relation to sound pedagogics for interaction design.
General Recommendationsnext section
Enabling a Performance-Driven, Dialogical Process
We have described, above, the benefits of using a pedagogical approach that emphasizes real time sound making and live demonstration of interaction scenarios, while also offering the possibility to challenge an interaction process with unexpected behaviors. In principle, both Foley and electroacoustic mockups can encourage exploration and ad-hoc ideation, especially at the beginning of the design process. These stages are akin to sketching and moodboards in visual design. But it is important to acknowledge that this method requires the participants to overcome inhibitions related to being on “stage” and, in addition, inhibitions related to making sounds in a playful way.
From our observations, the demonstrations were often a bit too controlled, and improvisation and ad-hoc ideation were not as common as we desired, as the participants followed their scripts, even in the Foley stage. This also has to do with how the educational system in general encourages the presentation of final concepts rather than the collective exploration of open ideas.
There are two strategies that can help to resolve these issues. First, it is important that the urge to “script” interactions in a restricting manner is resisted. The sonic experiences should always have a possible impact on the concept, and vice versa. Second, audience interaction, and the resulting moments of serendipity and surprise should be encouraged or even enforced. The presenters need to understand themselves as part of a rule-based, generative, and dialogical system rather than as actors acting out some previously prepared script.
Another concern is the (self-inflicted) urge of many students to “build” a functional prototype, as they consider this the only valid form of prototyping. The appreciation of non-functional prototypes and quick’n’dirty, low-tech methods is essential.
Enabling Sound-Driven Interaction Design
Providing methods that make use of the performance-driven sound design approach is the basis for enabling truly sound-driven interaction design. The “ease of use” of Foley and the seamless transition to the electroacoustic mockup allow the participants to use sound as material to design with from the beginning. The process facilitates the participation of all members in heterogeneous groups and enables non-sound designers to develop and finally implement convincing sonic ideas as well.
We see these sound-driven methods as an equivalent to conventional drafting and prototyping methods in interaction design. But these sound-driven methods are not established and are not always taken seriously. Therefore, “designing through sound making” needs to be advocated, and other more established methods for developing design ideas, such as sketching and storyboarding, need to be constricted, at least in the initial stage of development.
Avoiding a Purely Representational Use of Sound
If we are demanding an open and explorational approach towards using sound in interaction design, we are also avoiding the reduction of sound to a purely representational medium (i.e. indicating “error” or linear change of some kind of value). This reduction in our view is a central issue that prevents sound from being used successfully in many cases, as it leads to simplistic beeps and potentially annoying stereotypes that fail to stimulate interaction. The issue relates to the “stigma” of functional sound mentioned in section 4. In the case of the elevator scenario from Group 3, strangeness and surprising sonic qualities would have been promising paths to explore. We need to understand that expression can also be considered as function, and the character of a sound is part of its meaning.
Avoiding Stylistic Bias, Stereotypes and Trivial Mappings
From the case analysis we have seen that a design approach that is style-driven restricts possibilities for exploration of alternatives that could be more effective for the functioning of the design method. It usually leads to static and unresponsive solutions, because possibilities for modulations and variations are limited by the style’s formal requirements. In general, the flexibility and openness of the sound decline as aesthetic pre-determinations increase. Similarly, the simplification of sonic interactions leads to stereotypes and trivial mappings, which may clarify the interactions, but lead to dead ends in the pursuit of novel, fitting “sounds for tomorrow”.
A special case of stereotype is the human voice. While on the one hand it is a powerful means of sonic drafting, on the other, it requires extensive work to defamiliarize it, to detach it from its obvious human origins. In the case of Dipalu, this was not an issue, because the voice was meant to be the voice of the creature and could, as such, be sufficiently defamiliarized.
In some cases, the regress to predefined styles and stereotypes originated in a lack of sonic ideas, arising from an inhibition of exploration and a lack of experience in sound design, but possibly also a result of the representational approaches mentioned above. As it turned out, narrativity could be of great help in this case. Creating stories around artifacts as characters helped to define the sonic aesthetics and was also a useful basis to ensure an aesthetic coherence without being too restricted by a style.
Recommendations Related to Specific Stages
Establishing Foley as Demonstration Method
In order for the students to be able to use Foley for the design process required practice and minimal skills with the microphone. Otherwise, this stage could lead to an oversimplification of sounds, which leads to an unsatisfying quality of the demonstration, which in turn leads to abandoning an otherwise good approach. Live sound making as means of sonic interaction design needs to be taken seriously, and it is important to push participants to really explore the vast universe of sonic possibilities provided by objects.
Multisampling as Mediating Technology
With multisampling and MIDI mapping, this stage provides the basis where students are able to prototype their ideas in a more structured and controlled way than with the Foley method. Therefore, this stage requires a clear concept and, ideally, some existing sounds from the Foley stage to be effective. In the worse case, this leads to a rigid scripting of the interaction process and an emphasis on sequential triggering, in the positive case, the improved control over the sound production increases the expressive and narrative potential.
Most importantly, the electroacoustic live-mockup bridges the gap between Foley exploration and functional prototype, as it delineates in advance the relevant parts for the implementation as functional prototype, in particular concerning systematization of the concept and mapping of sound parameters to interactions and processes. The restriction on working with samples is useful for our target group, interaction design students, as it is consistent with the Foley stage. But in principle, the multisampler stage can be extended with other methods of digital sound generation.
Sensor Technology as Boundary
A prominent insight from the cases is the impact of the various technologies on the respective development of the projects. It is remarkable that in particular the difference between continuous and discrete acquisition of sensor data leads the students to either explore possibilities more openly, or to make restrictive decisions in a “yes” or “no” manner. The Dipalu case, for example, is based on a microphone as relatively imprecise input device, and thus motivated the students to think about situations as ambivalent and continuously changing. This ongoing exploration, coping with the technology and using it as malleable material, also contributed to the development of their concept. The opposite example can be found in that of the camera used for the realization of Sonotag. The existing functionality of the Microsoft Kinect camera was used to trigger events directly, and therefore limited the range of possible interactions. The group became more oriented towards functional implementation, which resulted in a much more defined and inflexible prototype. Also, the danger of the tracking breaking down motivated the use of a stereotypical and prominent warning sound which broke the narrative coherence.
Another point observed was that as soon as sensor technology is considered, be it conceptually or in actual implementation, it will take time and energy away from a sound-driven exploration. We strongly recommend starting with open methods for both sound generation and sensor technology and to maintain this openness as long as possible. This makes the process more undefined but leads to an extensive engagement until the end.