New Proposal

New Proposal - Focus on 3D visualiser

Proposal Project - MUS303 Interactive Sound Environments

Areas to be explored

This work will be exploring the ideas of data visualization, in particular with MIDI note information and audio. The MIDI information is going to be received from an synthesier and/or MIDI keyboard, and audio from either a music track or from the synthesiser/instrument. This may explore ideas around Toshio Iwai, with his “Piano - As Media Image” (Young, 2010). It will also look at performance based visualisation the same as Alva Noto and Ryuichi Sakamoto (Robin Chan, 2012).

Description of Project

For this project, I am proposing that I build a 3D visualiser that can be controlled with the MIDI out of a hardware synthesiser (MicroKorg). Elements that will also control the synthesiser will be the audio from the synthesiser and any backing tracks played at the same time. The visualiser will be mainly based on 3D shapes with each individual shape being controlled by a specific key on the synthesiser. Dials on the synthesiser, such as cutoff frequency, attack, and pitch bend, will control elements of the visualiser, maybe turning the shape or increasing the size. All the shapes will be of multiple colours.

What will this achieve?

When going to see live bands with keyboardists, they are often hidden behind keyboards, with little ability to express their skills with a visual indicator. This differs, for example, from a guitar player. A guitar player has his/her instrument facing the audience, and he/she can move about that stage freely to express the skills he/she possess. That being said, there is an instrument which can be used by keyboardists to reflect this very problem. A keytar. However, I found when watching live keytar players, that they are limited again to what they can do. Keytars are best played with one hand, which limits the skill of the player.

What the visualiser will achieve, is a visual indicator of the skill and expression of the keyboardist. Not only that, audiences enjoy live visuals during performances, and having shapes changing and moving with the music in time with the keyboardist can only increase the enjoyment of the show. Much the same way Alva Noto and Ryuichi Sakamoto do during their live performances.

So the aims of this project are:

• To create a 3D visualiser using Max MSP
• The visualiser should be controlled by a MIDI keyboard and/or hardware synthesiser
• Besides the MIDI information controlling the visualiser, the live sound coming from the synthesiser and/or a backing track should be able to control it.

Breakdown of how to fulfill this task

1. The first task will be to create a 3D visualiser that can be controlled by a prerecorded track. The visualiser should have multiple 3D shapes of multiple colours that pulse or change shape to the music.
2. Second task will be controlling the visualiser with dials on the synthesiser or MIDI keyboard. This will break the boundary of using the physical world to control the virtual world. The dials should be able to rotate the shapes or change the size/scale of the shapes.
3. Third, the shapes should be controlled by the individual keys on the synthesiser. So by pressing a key, it should change a specific shape by increasing the size of the shape. The size should be controlled by the velocity of the key pressed. Also, what would be good at this point, would be to have some element of having shapes controlled by a backing track at the same time of being controlled by the keys of a synthesiser.
4. The last task will be getting the shape's sizes to be controlled by the audio of the synthesiser, but also the key being pressed determines which shape is being controlled by the audio. It would be good to have the shapes move around the camera or the camera move around the shapes to give the visualiser different perspectives of the shapes.

Issues that I may come across

The biggest issue I will need to overcome will be getting individual shapes to be controlled by the individual keys on the keyboard. This is because there will need to a shape rendered for every key on the Microkorg and I will need to find a way to get Max to understand that I only want the specific key to change the shape. 

After being able to control shapes with velocity, it will be tricky to find a way to control the shapes with the the audio of the synthesiser and still have the keys mapped to the specific shapes. This will require multiple micro-patches and it's likely the Max Patch will get very big.

Being able to have a backing track as well as an audio input from the Microkorg could also prove to be difficult as it may be hard to seperate the two audio tracks from each other.

Bibliography

Robin Chan. (2012) Alva Noto Ryuichi Sakamoto Live in 2012. [Online] Available from: https://www.youtube.com/watch?v=8ggEYoVKX_M [accessed 13 November 2014]

Young, D. (2010) TOSHIO IWAI. [Online] Available from: http://www.inventinginteractive.com/2010/02/05/toshio-iwai/ [accessed 13 November 2014]