sound design & mapping

sonic experience

The sound content needed to be consistent with how people already perceive and experience the environment of the city. The scenarios of use we have developed + scenarios reveal differing personal relationships with the city. Specifically, we considered peripheral versus foreground aspects of the experience and musical possibilities ranging from ambient to rhythmical. As we wanted to maintain a close experiential relationship between the sound content and the context of music creation ­ (namely the existing city soundscape), we decided to use real time audio processing of urban sounds as a basis for the sound design. The musical output is shaped in real time by sound processing objects such as filters, delay loops, envelopes, sampling, playback, mute, and echoes. More information about the sound design and mapping is also available on the Viktoria Institute's Sonic-City project webpage.

MAPPING STRATEGY

The mapping had to be both transparent to the user and complex enough to sustain interest if the system were to be used day after day. We also considered it essential that the mapping would reflect scales of time and distances covered while walking in the city and maintain the distinction between continuity and discreteness of input. Sources of input selected during our urban observations + observations were therefore classified into two levels: low-level discrete and continuous factors coming directly from the sensors (see table), and high-level factors of general context and user actions resulting from abstractions of the low-level ones. In the mapping we developed, high-level abstractions of context and actions are mapped to structural composition parameters. Low-level discrete and continuous factors are mapped respectively the triggering of short musical events and to spectral variables. Thus, on a low level, rhythm patterns are algorithmically generated and sonically shaped based on sudden user actions and localised urban events and ambiances, while on a high level, the overall structure of the music is based on patterns of actions, path over time, and overall urban context. Within this general framework, decisions about details of the mapping were carefully made one at a time to insure coherence and pertinence.

classification of input   mapping strategy

VIDEO SIMULATIONS

We used video simulations as a method for supporting the sound design process. These simulations consisted of videos filmed by us in the city, to which we added soundtracks by inputing the video's audio into the prototype and by the sensors to simulate interactions suggested in the video. In those simulations, the following mappings were developed iteratively. Incoming urban sounds were processed through filters that opened and closed rhythmically at a tempo corresponding to the pace of the player while walking. The frequencies of these filters was determined by the intensity of light detected. The noise level of the local environment was mapped to the amount of rhythm in the music, such that rising volume increased the number of rhythmic layers that are overlaid on each other. Proximity to metallic objects or surfaces triggers brief echoes, the delay of which depend on the pollution level. At night, samples recorded randomly were echoed and filtered in relation to sudden flashes of light detected from street lamps.

+ video clip walking under a bridge 4,1M   + video clip climbing metallic stairs 6M   + video clip crossing a street 5,1M   + video clip walking under street lamps at night 2M

As an example, consider the following audio clip.

+ audio clip 288K

sonic experience