Controlling Visualiser with Keys and Amplitude

The last development post showed the visualiser being controlled with the keys on the Microkorg. However, the velocity made the 3D bars static and boring. The next step was therefore to find a way to route the audio of the synthesiser to each rendered bar.

One of the rendered bars. Main elements of discussion is circled

The objects that are circled in the image above are what allows the bars to be controlled by amplitude. The 'r k1' object is the receiving the velocity of the corresponding key being pressed. The 'r band1' is receiving the amplitude. These are both being sent to a multiplication object which is then being fed to control the scale of the 3D bar. This works because even if the amplitude being sent is high, unless the key is being pressed sending a velocity number, the amplitude is being multiplied by 0, so it's equal to 0 meaning no scale increase. However, what I found was because the velocity went up to 15, as it was being clipped at 15, the scale got pretty big when being multiplied by 15. To fix this, I changed the pong object so it was now being limited to 1. This kept the scale under control and looking good.

Screenshot showing that velocity is limited to 1 with pong object.

The keys were now doing exactly what I wanted this project to do now. Each key controlled a corresponding 3D rendered bar, with the scale being dictated by the amplitude of the audio. It looked brilliant. But I wanted to make it look a bit more interesting. I was inspired by how another project was spinning as it was showing it's visualiser. See blog post Other Interesting Max MSP Visualisers . I also wanted to add another element that would allow a backing track to be represented with a 3D rendered shape. This would be the next step that will hopefully finish the project.

Other interesting Max MSP Visualisers

As my project is entirely based in Max I did a lot of searching around before building my own visualiser. Although mine will never be on the same par as with these projects, these other visualisers did inspire me to what the capabilities were of using Max and Jitter. It was good as an approach to think "if I aim for moon, I'll only get to the clouds, but if I aim for the stars, I'll get to the moon". In relation to the my project, if I aim for the best project, it will turn out much better than if I aim for it to be just as good as it can be. 

Above is a visualiser that uses frequency to create visuals. This was one the ideas that inspired my project to be controlled by frequency too. Although it was very difficult to work out. However, aiming for this project did lead me to learn how to read amplitude of audio, which was one of the fundamentals to moving the scale of my objects using amplitude.

No real features of this visualiser really made it into my project, however this was one of the first visualisers I came across that was controlled by Max. So this showed what could potentially be the maximum potential of what my visualiser could be. I like how fluid this visualiser is to.

This project is more like my project, in the way that it looks and works. But what relates this most to my project is the way it is controlled using hardware. This is using the Toshio Iwai theory of breaking the boundary between physical world and virtual world by using an interface to control the visuals. I like the way this project is always spinning, which is why I want my project to always spin as it keeps the visuals looking interesting rather than static.

This visualiser is one of many made by someone called Synzou Nagai. His visualisers are amazingly complicated and produce some beautiful looking visualisations. What makes his projects amazing is that looking at the MAX patches, they don't look complicated. But once you begin reading into how he uses the objects with recorded videos of lights and objects they get incredibly complicated. These visualisations don't compare that well to my project, as there is no external control, however the way it is controlled with frequencies and amplitudes makes the visuals look spectacular. His websites shows even more of his projects.

http://synzosizer.com/m_m.html

Carsten Nicolai/ Alva Noto

Carsten Nicolai, also known as Alva Noto, creates a lot of Data Visualisation projects. His works span over quite a few subjects, but one of the subjects I found interesting was his unicolor, unidisplay, and unitxt installations. These installations are a series of projects designed to great ever expanding moving images of colours and shapes, some of which were created with audio hardware.

Unicolor Installation

Nicolai creates these ever expanding images using mirrors. When actually viewing the Unicolor installation, it can confuse the viewers perception as the images are always changing and they expand infinitely due to the placement of the mirrors. This is to create an impression of an ever expanding universe. This can relate to my project, as the rendered images are in an ever expanding 3D plane. The way the 3D shapes work in my project, they can go off the screen, particularly when the pitch is bent, to create this same effect of an ever expanding universe.  To try and relate Nicolai's work to mine, I may try and find a way to spin my project continuously, to show how big the 3D space really is, to create this perception of an ever expanding Universe.

A project made by Nicolai actually helped inspire my project, when it was the LED piano, but still relates heavily to my continuous project. This was a live show of Carsten Nicolai working with musician and composer Ryuichi Sakamoto, to create this data visualisation along with a music performance. Sakamoto is on the piano and Nicolai is using a laptop and together they create this ambient music. However, what makes this performance more interesting is the data visualisaion projected behind them. The data visualisaion brings more life to their performance and creates more meaning to the music. What makes the visuals so effective for them is that they are subtle. The visualisation doesn't distract from the performance, it enhances it. This is exactly what my inspiration for this project was, particularly as I consider myself an ambient composer.

References

Nicolai, C. (2015) Unicolor. [Online] Available from: http://www.carstennicolai.de/?c=works&w=unicolor [accessed 21 March 2015]

Robin Chan. (2012) Alva Noto Ryuichi Sakamoto Live in 2012. [Online] Available from: https://www.youtube.com/watch?v=8ggEYoVKX_M [accessed 13 November 2014]

Control with keys

The next step was to try and get the individual keys on the Microkorg to control each cube in the visualiser. In my mind I knew how this could be achieved, but actually tackling it was going to be difficult.

I began by experimenting with the MIDI related objects in Max, loading in different objects and using the help files to explain what each did. I soon discovered that the midiparse object will print everything that is happening to a MIDI controller. This included: note being pressed, velocity, dials being turned, how much the dials were being turned and more. What made it interesting for me was what it could offer my project. I noticed, if I played a chord, for example C, E, G, even if I played those notes at the exact same time, it would produce the messages for each key, including velocity. Plus, when I removed my fingers, it would send messages to show that the keys were not being played, as it would send out a 0 velocity for each key. This lead me to the idea of having separate micro patches that I could somehow teach to only send out a message, when the corresponding key was pressed, to change a corresponding 3D bar.

Above is a screenshot showing part of my prototype patch that allowed individual key control. Each object send is being sent to an unpack object which allows the MIDI signal to be recognised. I had 37 send objects and 37 recieve objects, 1 for every key on the Microkorg. The next part was very difficult. I knew, from coding in the past, that I could use an "if" statement to filter out the other MIDI key messages to leave only the MIDI key I wanted. After much trial and error, and much help from the max help files, I figured out the "if" statement. Below is an example code that allows MIDI key 48 to filter out so that only, if the MIDI key 48 is pressed, it will allow through.

if $f1 == 48. then out2 $f2. else

Basically what this means is: if the floating number coming into port 1 is received and it is equal to 48, then send out of port 2, the velocity. If the signal coming into port 1 isn't equal to 48, then do nothing.

After I wrote an "if" object for every MIDI key, I moved this patch to my project patch and sent each one out to their corresponding 3D rendered bar. I then routed them so that the velocity would control the scale of their corresponding bar. However, what I found was that because the velocity could be anything between 1 and 127, the scale was so big it would go off the screen. To limit the signal, I used a pong object to limit velocity to 15. Again however, this meant that normally the velocity was 15, making the 3D bars static and boring. Therefore, my next step of development was going to be finding a way to use the amplitude of the audio of the synthesiser, to control each 3D bar. Below is a screenshot  showing part of the patch with the pong objects and the recieve signals controlling the scale of the 3D objects.

Toshio Iwai - Main Inspiration and Practioner

Toshio Iwai is an installation artist who focuses on creating interactive installations, mainly relating to sound and visual. One of his best known installations is the 'Piano - As Image Media' project. This project used a trackball to create visual bars of light that was in sync with sound. This was the primary inspiration of my project, as the theory of Iwai's work is breaking the boundary between the physical world and the virtual world through data visualisation. My project uses this same theory by using the physical world, the synthesizer, to control the virtual world, the visualiser. I also used the same idea of the horizontal streaks of light that represent the keys being pressed from Iwai's installation; in my project I have horizontal beams that represent the individual keys of the synthesizer.

Piano - As Image Media Installation. 

Another work by Iwai that uses a similar theory, of breaking the boundary between the physical world and the virtual world, is his Composition on the Table installation. This installation uses projected images on a table that can be controlled with your hands to create music and sounds. In some ways it actually relates more to my project as it has different elements to control sound, such as dials and switches.  Sounds are produced in relation to the way the projected images are moved. Essentially my project is a mirror image of this being that his project uses controlling the virtual images to create sounds, my project uses sound to control virtual images.

Composition on the Table installation.

References

Young, D. (2010) TOSHIO IWAI. [Online] Available from: http://www.inventinginteractive.com/2010/02/05/toshio-iwai/ [accessed 13 November 2014]

technOasis. (2000) Toshio Iwai. [Online] Available from: http://www.siggraph.org/artdesign/gallery/S99/big/artists/Toshio_Iwai.html [accessed 19 March 2015]

New Proposal

New Proposal - Focus on 3D visualiser

Proposal Project - MUS303 Interactive Sound Environments

Areas to be explored

This work will be exploring the ideas of data visualization, in particular with MIDI note information and audio. The MIDI information is going to be received from an synthesier and/or MIDI keyboard, and audio from either a music track or from the synthesiser/instrument. This may explore ideas around Toshio Iwai, with his “Piano - As Media Image” (Young, 2010). It will also look at performance based visualisation the same as Alva Noto and Ryuichi Sakamoto (Robin Chan, 2012).

Description of Project

For this project, I am proposing that I build a 3D visualiser that can be controlled with the MIDI out of a hardware synthesiser (MicroKorg). Elements that will also control the synthesiser will be the audio from the synthesiser and any backing tracks played at the same time. The visualiser will be mainly based on 3D shapes with each individual shape being controlled by a specific key on the synthesiser. Dials on the synthesiser, such as cutoff frequency, attack, and pitch bend, will control elements of the visualiser, maybe turning the shape or increasing the size. All the shapes will be of multiple colours.

What will this achieve?

When going to see live bands with keyboardists, they are often hidden behind keyboards, with little ability to express their skills with a visual indicator. This differs, for example, from a guitar player. A guitar player has his/her instrument facing the audience, and he/she can move about that stage freely to express the skills he/she possess. That being said, there is an instrument which can be used by keyboardists to reflect this very problem. A keytar. However, I found when watching live keytar players, that they are limited again to what they can do. Keytars are best played with one hand, which limits the skill of the player.

What the visualiser will achieve, is a visual indicator of the skill and expression of the keyboardist. Not only that, audiences enjoy live visuals during performances, and having shapes changing and moving with the music in time with the keyboardist can only increase the enjoyment of the show. Much the same way Alva Noto and Ryuichi Sakamoto do during their live performances.

So the aims of this project are:

• To create a 3D visualiser using Max MSP
• The visualiser should be controlled by a MIDI keyboard and/or hardware synthesiser
• Besides the MIDI information controlling the visualiser, the live sound coming from the synthesiser and/or a backing track should be able to control it.

Breakdown of how to fulfill this task

1. The first task will be to create a 3D visualiser that can be controlled by a prerecorded track. The visualiser should have multiple 3D shapes of multiple colours that pulse or change shape to the music.
2. Second task will be controlling the visualiser with dials on the synthesiser or MIDI keyboard. This will break the boundary of using the physical world to control the virtual world. The dials should be able to rotate the shapes or change the size/scale of the shapes.
3. Third, the shapes should be controlled by the individual keys on the synthesiser. So by pressing a key, it should change a specific shape by increasing the size of the shape. The size should be controlled by the velocity of the key pressed. Also, what would be good at this point, would be to have some element of having shapes controlled by a backing track at the same time of being controlled by the keys of a synthesiser.
4. The last task will be getting the shape's sizes to be controlled by the audio of the synthesiser, but also the key being pressed determines which shape is being controlled by the audio. It would be good to have the shapes move around the camera or the camera move around the shapes to give the visualiser different perspectives of the shapes.

Issues that I may come across

The biggest issue I will need to overcome will be getting individual shapes to be controlled by the individual keys on the keyboard. This is because there will need to a shape rendered for every key on the Microkorg and I will need to find a way to get Max to understand that I only want the specific key to change the shape. 

After being able to control shapes with velocity, it will be tricky to find a way to control the shapes with the the audio of the synthesiser and still have the keys mapped to the specific shapes. This will require multiple micro-patches and it's likely the Max Patch will get very big.

Being able to have a backing track as well as an audio input from the Microkorg could also prove to be difficult as it may be hard to seperate the two audio tracks from each other.

Bibliography

Robin Chan. (2012) Alva Noto Ryuichi Sakamoto Live in 2012. [Online] Available from: https://www.youtube.com/watch?v=8ggEYoVKX_M [accessed 13 November 2014]

Young, D. (2010) TOSHIO IWAI. [Online] Available from: http://www.inventinginteractive.com/2010/02/05/toshio-iwai/ [accessed 13 November 2014]

Change in Project

As it was becoming more difficult to work on both the 3D visualiser and the LED keyboard at the same time, I have decided to drop the the LED keyboard project and put all my focus into the visualiser. I chose drop that project because to create my visualiser, I am using Max MSP and Jitter, which is something that I already have a good understanding of and, being that it is mainly computer based, it would be easier to work with. The LED keyboard however, because I was using an Arduino, there was a big learning curve to understand the code in that. Not only the code, the electronics side was another learning curve. Plus, being able to have 24/7 access to an Arduino was difficult to have unless I purchased one for myself, which I did not have the financial capability to do.

I also worried that my LED keyboard might not work or that it wouldn't be finished in time for the assignment deadline. Furthermore, I was also aware that the LED keyboard on it's own was not to a large enough scale that would be interesting to a live audience; so if I was to drop one project or the other, it was always going to be the LED keyboard, regardless of how much of a learning curve there was. I do have a desire to finish this project, but it will have to be in my own time. 

As a result of this new focus, I have written up a new project proposal that focuses on the visualiser. This will be in a post after this one.