My notes from Zach‘s Making the invisible visible, a talk that he also dubbed “How i came to loose all of my hair in the last 5 years.”

“I travel a lot and never know what i should write or say when i’m asked about my occupation. “Artist” sounds too egotistical, so i’d prefer to define myself as a “researcher” as i feel that my artistic practice is a kind of research. I use technology to augment the body, the intellet and extend ourselves. Unlike other instruments, like guitars, computers are a relatively new tool, we’ve just started to explore their creative potential.”

Together with Golan Levin, he has created TMEMA. A fisrt series of their projects revolves around the voice.

The Hidden Worlds of Noise and Voice (2002) was influenced by one of Czech animator Michaela Pavlatova‘s works Reci Reci Reci (means “Words, words, words”). The drawings make visible the words that goes out of the characters’ mouth.

The augmented-reality speech-visualization system reveals the relationship of speech to the ethereal medium which conveys it. Each of the six persons around a table wears special see-through data glasses (equipped with microphone and position sensor), which register and superimpose 3D graphics into the real world. When one of the users speaks or sings, colorful noodles appear to emerge from his or her mouth. The graphics are tightly coupled to the unique qualities of the vocalist’s volume, pitch and timbre.

That same Summer they made a second project called RE:MARK. This time their enquiry is shifted towards a more symbolic representation of the spoken and written word. Sounds spoken into a pair of microphones are analyzed and classified by a phoneme recognition system. When a phoneme is recognized, the written name of the phoneme (for example, oh, ee, ah, etc.) is projected on the installation’s display. If the user’s sound is not recognized by the system’s classifier, then an abstract shape is generated instead, based on the timbral characteristics of the vocalization.

As the visitor speaks, the phonemes and forms appear to emerge from the shadow of the speaker’s head. A computer-vision system permits the visitors to interact with the sound fourms by using the shadow of their own body.

After that Levin and Lieberman were wondering what the next step could be. The works were quite successfull in a museum context but what would a professional vocalist do with such instruments? They worked with Jaap Blonk and Joan La Barbara on a 40-minute long performance called Messa de Voce. Not a single word is uttered but the whole performance was about language and communication. A software transforms every vocal nuance into correspondingly complex and highly expressive graphics. While the voice-generated graphics become an instrument which the singers can perform, body-based manipulations of these graphics additionally replay the sounds of the singers’ voices — thus creating a cycle of interaction that fully integrates the performers into an ambience consisting of sound, virtual objects and real-time processing.

Liberman then briefly talked about Drawn, a performance/installation in which ink forms appear to rise off the page and interact with the very hands that drew them. The work explores the musicality of drawing by turning brushstrokes of ink into complex and energetic life forms. Video.

Zach is currently looking into ways to use these technologies for in the classroom for children with profound multiple learning disabilites as part of a NESTA education grant. He’s working on a low cost version of the installations, designing a software that uses motions as an input to interact with the children. The software is adjustable to have a relationship with the children. So far the kids like it, it’s an empowering experience for them: they are unable to make big motions but they can see that they are able to control the whole installation.

Related: Golan Levin keynote at Cybersonica in 2005.

Interview of Zachary Lieberman: pintar la voz. Scroll down for english version.