d) method


The gaming industry provided us with inspiration and technology that we could use as a starting point for the physical interaction. Other sources of inspiration came from sensors developed for measuring athletic performance and from mobile phones. We combined different aspects of these together with artistic expression to design various types of interfaces.

We tried to follow two set conditions;

a) “No learning curve” and

b) “Intuitive”

We aim at giving the interfaces their own identity or “soul” which we believe inspires positive curiosity and encourages exploration.

By using already existing computer programming environments for real time generated audio and graphics we could create a base platform for the audio/visual design.

The audio/visual output is interpreted and analyzed in terms of emotional content and expression. These results are then being compared with the users intention. This relation is the core part of the research, – to what extent the user feels that her intention and experience of the action matches the artistic expression of the audio/visual output. To achieve this, the raw interaction data has to be interpreted in order to affect the artistic expression of the audio/visual output in a relevant way.

As an inspirational and pedagogic model we studied the art of calligraphy where movements can be interpreted dynamically over time. A trained eye can, in a calligraphic drawing, read factors such as brush approach, turns, speed, temperature and pressure and determine the emotional content of the outcome.

How can this be achieved by machines?

Calligraphic drawing by Jonas Ericsson

Factors and interpretation recipes

The users interaction with the interface is recorded and constantly updated in real time by accelerometers, various types of sensors and cameras. In order to receive information that is as versatile as possible we need to separate the data from the different event types in the interaction. Examples of event types could be pressure, acceleration, velocity, intensity, regularity etc.

A program that identifies peaks in a signal and the interval between them. A signal can for example be transmitted from a sound source or a sensor. The data is used both to measure the tempo and regularity. The regularity can describe how rhythmical a signal is.

The collected data is fed into “interpretation boxes”, a matrix device where the data from the readings affects the output parameters according to a set recipe. Recipes in this context are a way to define to what degree the interaction data affects different aspects of the audio/visual output. For example, to what degree the intensity of a movement affects the audios “sharpness”, or the visuals “color temperature”, or to what degree the derivative character of the data affects the tempo of rhythmical elements in the audio or visuals.

Different interpretation boxes can accumulate events over time in order to make the system more complex and dynamic.

Snapshot from interpretation matrix used to create recipes.


Initially, we have created a number of “scenes” as experimental platforms. We have chosen to use abstract and dynamic audio and visuals, rather than tonal and illustrative. The aim is to enhance user responsiveness. If she does not recognize what she hears or knows what to expect, her participation will be more open and intuitive.

Haptic feedback provides another mean by which the user can be stimulated and led thru the experience. Rules for how dramatic events develop can be created.

Another important element in the design of the system is that relationship to our ability. A four or five year old sitting in front the audio/visual output never should repeat itself in order to of a piano or a drum kit rarely feels limited by what could be stimulate the user to explore the scene again and again.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: