“Monalisa: shadow of your sound,” by Norihisa Nagano and Kazuhiro Jo, is an installation that allows one to interactively “watch sounds and hear images.” It’s currently exhibited at the renovated ICC Museum in Tokyo (if you want to know more about the museum’s recent happenings, see the PingMag article “Rebirth of Japan’s Media Arts Centre ICC.” I (manekineko) recently went into Monalisa to hear the sound of my face. Then, I sang to it to modify my face.
The main component of this interactive installation is the software application called Monalisa that melds sounds and images using Monalisa-Audio Unit and Monalisa-Image Unit plug-ins. The audio unit processes sounds using image processing mechanisms and the image unit processes images using sound processing mechanisms – for example, one can rotate a sound sequence using an image rotation tool and likewise one can manipulate an image’s frequency levels using a sound processing tool.
[Monalisa: shadow of your sound.]
I asked Kazuhiro (K) and Norihisa (N) questions to know more about the ideas behind Monalisa.
What motivated you to create Monalisa? I could view Monalisa as a machine that produces very unique sounds and images. I could also view it as an environment that provides opportunities to think about the image-sound relationship.
K: I surely want people to hear those sounds and the images, but one of the intentions of showing it as an installation is to allow people to think about the relationship between image and sound through interactive experiences.
By rolling out the image-sound-image conversion process in a physical space, we can let users see and hear parameters; I mean the parameters for changing characteristics of cameras, speakers, a room, and microphones. They are used to transform images to sounds and then to images. In my opinion, this seems interesting as a method for presenting media, visually and audibly.
[image taken from the Monalisa brochure.]
Were you aiming at making a new musical instrument? Is Monalisa designed as a musical instrument that users can play with fluency?
K: When we made Monalisa, we thought about its aspect as a tool, perhaps like a musical instrument or a paint brush — not just as an art installation or a machine to make people think. But it’s made as a specialized tool for possible user population, maybe one in ten thousand people – it’s not made as a tool that anyone can use fluently.
N: “Bridge,” “method,” and “anti-these” are the keywords for thinking about what Monalisa is. We can call it a musical instrument or a tool since it makes sounds, but it is also a “method.” That is, Monalisa is a bridge that connects images and sounds – they are treated differently even though they are both internally zeros and ones on a PC. And Monalisa provides a bridging method. Also, it’s my own way of showing the inevitability of using computers for reasons beyond “computers are useful”
It’s very interesting that Monalisa converts images to sounds and then back to images. This makes me think about an infinite loop that repeatedly merges bits and atoms in a Jimi Hendrix style. Along this line, is Monalisa designed for personal, immersive experiences rather than multi-user, social experiences?
K: It doesn’t really matter if it’s used by a single person or multiple people, but we think it’s important that visitors intervene in Monalisa’s conversion process in the physical space, and don’t just passively view it. In this sense, it can be more about immersion than sociability, but we haven’t set any rules about what’s right. I also want to see and hear intervention methods we didn’t anticipate – not just shouting or covering the microphone by hand.
What are your future plans?
K: We are making the Monalisa software available for download. Currently Monalisa-Audio Unit and Monalisa-Image Unit plug-ins are available and the Monalisa’s main application will be released soon. We also plan to update the installation at ICC.
N: In the future, I’d like to show the “method” that the three software applications enable.
Any other comments?
N: In the Monalisa installation, what people see and hear are the same. It’s not just synchronization of sounds and images. Reflecting on this situation of seeing and hearing the same thing at the same time, I’d hope to gain some hints for the next ideas about what computers can do.
You can download the plug-in software at: http://monalisa-au.org. (Nao Tokui also has some audio samples made using Monalisa.)