What does it mean?

‘Another Thrill in the Wall’ is our motto

Where thrill refers to that thing you feel while an emotion is spreading through your body. The AffectiveWall is all about your body and how you can use it for affective expression! Imagine a multitouch surface that captures affective states and emotions, manifested by whole body interaction as if you are painting a big canvas, and translate them to music and digital painting projected on the wall in real-time.

We follow the hint of Leopold Stokowski,

a painter paints his pictures on canvas, but musicians paint their pictures on silence

Coming up with an instrument that mix both, musical and visual creation (and consequently being also a tool for performance), to be played in an organic and intuitive way (with the lowest possible learning curve), enabling its use in Arts and also for educational and therapeutic purposes.

Why we did an instrument like this?

  • To be easy and intuitive, enabling its use by any person regardless of his/her background
  • To use visual feedback to help you compose music and, at the same time, to allow your audience to better understand what you are trying to express
  • To be controllable with your hands or other body parts, without the need of mouse, keyboard, controllers or other stuff
  • To always generate feedback, decreasing your frustration while trying to learn how to use it
  • To be used in mixed-art contexts like installations and performances, or as a tool for composing music in experimental and electronic music fields
  • To be used in a therapeutic context due to ‘tactile’ contact with sound. Well, maybe. For the sake of honesty we still didn’t try it with children with disabilities, but it’ll be interesting to experiment as an art therapy towards the exploration of their creativity and communication skills
  • To be an experience of sound and vision that mixes freedom, expression, creativity and, ultimately, fun!

What inspire us for doing this stuff

From music we took the emotional expression of romanticism combined with the sound style of modernism movements and experimental electronic music

From painting, the symbolisms and subjectivity of abstract art, mixed with the focus on the act of painting of action painting, the impulsivity of expressionism and the elevation of the art produced without artistic knowledge of art brut

From mix-media art, the connection between different fields of art, like music and painting, of artistic movements like intermedia, fluxus, lumia, etc.

Multidisciplinarity due to being a tool for creativity through the stimuli of linking different artistic processes to promote novel contents (e.g. composing music through the act of painting, or create painting according to the sounds generated)

Synesthesia due to being a neurological phenomenon in which stimulation of one sense leads to the stimulation of another (e.g. visualizing colors while listening to music, or having sound experiences while seeing paintings)

Improvisation due to being a tool for intuitive expression and consciousness expansion, creating art by following our impulses (more emotional than rational!)

“Truly fertile Music,
the only kind that will move us (…)
will be a Music conducive to Dream (…)
One must not wish first to understand
and then to feel.

Art does not tolerate Reason.”

Albert Camus

Art is all about expression.

And if emotions and body language can be universal ways to communicate, a musical and painting instrument that uses affective expression through gestures as its interface emerges as a way of bringing Art closer to people. AffectiveWall aims to put all these stuff interacting together:


  Affective states (mood, emotions), it’s why we want to communicate, it’s our need of emotional expression


  Gestures/body-language, it’s how you create something, playing by simple touching with your hands or whole body on the surface


  Music/sound and digital painting, it’s what will be created, the final result after the interpretation of your emotional expressivity


  Wall/canvas, it’s the surface where the creative process and the visual/sound production takes place


  In real-time, improvising and following your deepest intuitions while freely experimenting with colors and sounds


 You and everyone else, regardless their background or if they have artistic knowledge

The way all these are connected to create an artistic and affective experience is by a performer – like you – touching the wall/canvas (a vertical multi-touch interface) that analyzes features of your gestures, the way you move and ‘paint’. This analysis feeds an Affective Ecosystem Interface which decides the affective states expressed according to our Affective Model for Gestural Expressivity. With the affective states perceived, AffectiveWall composes music and abstract paintings as a way to better express the performer’s Augmented Affective States.

How it works?

Affective Ecosystem Interface

In the beggining was the gesture.

AffectiveWall uses a LLP setup as the multitouch surface, and the gestures are read by a camera and translate to useful information by CCV tracking software. Therefore, using TUIO communication protocol, these informative features feed our Affective Ecosystem, developed in OpenFrameworks.

This concept is inspired by ecosystem interfaces, where is the system that has the main role on the interaction, responding to the environment that can be altered by users (but also by space conditions, noise, etc). In this way the users can indirectly interact with the system by actions on the environment.

AffectiveWall relates to this concept by having an ecosystem of affective states as the interface, where the system has the role of interpreting them, and the performer the chance to change them (therefore indirectly change the system output).

In this way we aim to achieve

A non-limitation of interaction options (no limited choices on a menu!) leaving the interaction to a continuous sequence of controls (free gestures and movements related to the act of painting

A separation between the interface and the sound/visual production, creating an affective interface instead of a physical one (limited by guitar strings or piano keys) to avoid the performer to be worried about producing low-level contents – like chords – but rather focus on how to represent a specific affective state that s/he wants to express

Affective model for gestural expressivity

For creating this Affective Ecosystem, we ask the help of the live painting artist São Nunes and a bunch of awesome people that were willing to test our ideas, and create an aesthetic/artistic approach model for converting gestural expressivity in affective states. We did that by evaluating people while they express themselves on a wall, in order to find gestural patterns. The affective states we evaluate were

sadness  shame  anger  confusion  joy
freedom  melancholy  pride  pleasure  exaltation
tenderness  shyness  satisfaction  loneliness  hate
fear  relief  hope  disappointment

By evaluating their gestures while expressing themselves we could understand some patterns of how people (both from artistic and non-artistic backgrounds) tend to transform these affective states in actions through their bodies (analyzing both their impulsive expression and the expression after reflecting which gestures they want to use for each affective state).

By measuring things like

gesture length     area of touch     quantity of gestures
gesture speed     direction of the movement     shapes drawn
duration of the gesture     location on the canvas
occupation of the canvas

We ‘teach’ AffectiveWall how to perceive what the performer is trying to convey while playing (a.k.a. kind of like painting on a wall/canvas). So, for instance

Sadness is typically composed by one slow and sustained gesture drawing a downward line starting at medium height and ending near the floor, using from a single fingertip to almost the whole body slowly sliding along the canvas

Anger is commonly represented using both hands at medium height to make punctual and static gestures like punches, or fast and sustained movements like ripping the canvas

Joy is most of the times equal to many, long and sustained gestures at medium or high speed, performed at medium or high height and using a lot of space in the canvas, like excitingly and randomly rubbing as most canvas as possible

Tenderness has high probability of being express with one long, sustained and slow gesture, drawing a line at medium height with medium occupation of the canvas

Then, we follow some studies about the connection of emotions and colors/sounds (references on the documentation in the end of this page) and came up with things like this

An regarding sound qualities, for instance if anger is more perceived with music with fast tempo, many harmonics, high frequencies with few variations, high volume, staccato articulation, distorted timbre, sudden attacks and noise; tenderness is more related to music with slow tempo, few harmonics, soft envelope, low volume with few variations, legato articulation and soft timbre.

So these are the kind of visual and sound information that compose the output – or what we like to call – the performer’s Augmented Affective States.

Augmented Affective States as output

After perceiving what the performer is trying to express, AffectiveWall composes visual and sound feedback in real time with the output of the Affective Ecosystem. This was programmed using OpenFrameworks for the digital painting and SuperCollider for the sound synthesizers, and the result consists on the Augmented Affective States of the performer. This is the same as saying that what the audience perceive is not only one output (for instance sound) but rather a whole combination of sound, vision and physical gestures and movements of the performer (almost like a dance performance!)

A resume of this and, therefore, the process of communication between a performer and an audience is something like this

AffectiveWall delivers this multimodal stimulus as a way to help an audience to better perceive the affective expression of the performer. And, of course, also to create an immersing environment that fosters an intimate relationship between performer and audience. Possibilities are endless and, for instance, applying AffectiveWall in a whole stage with multiple dancers using their bodies in a choreographed or improvised way to build-up an intermedia show can be one of many applications of our instrument!

These are some users testing it in a small canvas

And you can also see one more experiment in this video…

Resuming, this is how the stuff happens…

In the end of day, we really like J.Han’s words – “multi-touch-sensing was designed to allow non-techies to do masterful things while allowing power users to be even more virtuosistic” – so, with AffectiveWall, we are interested in applying this to Arts, in order to simplify them to universal languages known to everyone: emotions and body gestures.

Because, all in all, it’s just another thrill in the wall!

AffectiveWall was a master thesis done by Miguel Jerónimo with coordination of Carlos Martinho and Ana Paiva. You can download the thesis here or have a look to a paper that kind of resumes it in a couple of pages here.