Finished Project - Download Max Patch here, Instructions for use, Demonstration Video

Below is the link to my completed Max MSP Visualiser Patch. When copied into Max, I suggest putting it into Presentation mode as the instructions below will be easier to follow if used in presentation mode. 

https://docs.google.com/document/d/1ZRgQLCmuHPSCkWyAhdbRv_nud2ZXqxnMCevEIVKHs4M/edit?usp=sharing

Instructions on how to use:

1. Before opening patch, make sure Synthesiser (ideally a Microkorg) is plugged into the MIDI IN port on USB interface. Check Max settings that MIDI information is coming into Max and make sure MIDI is coming out of the MIDI OUT port on the synthesiser. 

2. Open Max project. Change to presentation mode and keep patch locked.

3. Render the Visualiser by clicking the toggle labelled below as "Render Visualiser", this will also load the presets. You can press escape if the visualiser goes full screen to see controls again. 

4. Press >Enter< to enable audio to digital conversion. This is necessary for Max to hear your synthesiser.

5. Begin playing. If set up correctly, 3D bars should react to the keys you press on sythesiser. If nothing happens, try pressing keys up and down through different octaves as this program was written primarily for Microkorg which only has 37 keys. If still nothing, make sure Audio In is enabled as Max might not be able to hear your Synthesiser. If still nothing, it the visualiser rendered? To check this, is the Render Visualiser Toggle pressed? (see image below). If still nothing, is the volume up on your sythesiser/interface? 

6. As you play, you may notice that the visualiser is spinning. If you wish to stop this, press >S< on computer keyboard. If it isn't spinning, and you wish it to, press >S< on computer keyboard.

7. To change colour of background in visualiser, click the colour bar in the interface labelled (erase_color). Use the colour wheel or presets to changes colour, the background of visualiser should change automatically. You can either close window after or leave open to change whilst playing. 

8. To load backing track or song, press >T< and select desired track on your hard-drive. To loop the track, press >L<  NOTE: backing tracks only work with AIFF files.

9. If you do not wish to use hotkeys, you can use the interface controls to do all of the above, each control is labelled. 

Demonstration Video

Below is a demo video showing the finished visualiser being used. This video was shown during my presentation. What it presents is the actual visuals being controlled with the Microkorg in a few different styles of music and different ways.

Final Thoughts

The project is now finished so now was the time to think about what will become of the project.

Who is it for?

If I had to give my project a 'target audience' it would be for electronic artists, but probably more towards ambient/chilled house artists. This is because the music that is ambient and chilled house is quite mellow, so having something like a visualiser behind them whilst performing would help enhance their performance, much the same way Alva Noto and Ryuichi Sakamoto used the visualiser behind them when they did live performances. My original vision however didn't have a particular target audience, I wanted it to be fairly open to whoever wanted to use it. It could even be used by a keyboardist in a rock/metal band. 

Will I distribute it?

This blog will act as a distribution for my software, being released as 'Version 1' this gives me a chance to improve on the product and blog, possibly leading to a 'Version 2' or 1.1.

What could have been improved?

What could be improved is it's usability. The patch itself is very easy to use, once it's in presentation mode, so giving the user more control would be a good improvement. This could be features such as having the ability to change the colour of the shapes or change the shapes themselves. It possibly would be nice to have a much more refined look for the patch, however I like how minimal it looks, so it possibly would be good to make it look more professional looking.

Another feature that I would like to add in the future would be creating different ways to control the visualiser. So this could be by allowing a DAW, such as Reason or Ableton, to control the visuals, or even a whole other instrument such as a guitar. From a technical point of view, what could be improved is that in the visuals there are these floating squares for where the 3D strips will be when played. Finding a way to make those invisible would be better for the viewer to look at it, especially in a performance point of view.

Presentation - Slides and Tutorial Video

Here is my presentation slides that I used in my presentation.



Below you will find a tutorial video explaining how my Max Patch Works

Final Touches

The project is now at a point I would say it is complete. Although there is always ways to improve a project like this, it is now at a point where I can say it is finished. All the elements work well together and the visuals look great. Finishing touches is all I needed to do now.

Finished whole view of patch

This is how the patch looks. I have used panels to sepearate the elements into different sections. The green panels are all the controls and inputs etc for the 3D rendered bars. The blue on the right of the patch are the 3D rendered spheres. The pink at the bottom is the controls to separate out the MIDI In key messages. The top includes all the various controls for the sythesiser.

What I have added since the last post is various controls for the visualiser. It now includes a control to select a different background colour for the visualiser. There are now hotkeys to control elements. Around the project there are a few comments on what each part of the patch does. Finally, there are instructions at the top to how to use the visualiser.

After adding in these elements, I decided to create something in presentation mode so that there were minimal controls for the visualiser. This keeps it easy to use and it's so almost anyone can use it without any knowledge of how Max works.

Presentation Mode with minimal controls

Backing Track Sphere

In my proposal I wanted to have a way of representing a backing track should someone wish to have one whilst playing the synthesiser. I incorporated one using as a rendered 3D sphere that pulses to the music. I did this by having the same setup as a 3D rendered bar, but changed the shape to render as a sphere. However, I did have to use a scale object to limit how much the sphere pulsed as the sphere did get very big if the song was too loud. By limiting the scale of the amplitude, it made the pulse much more under control.

After incorporating the backing track sphere, I realised it looked strange only having 1 sphere. To help keep the balance, I added another sphere, however this was controlled by the amplitude of the sythesiser. This worked well with the 3D bars, as sometimes the synthesiser audio would have some decay; but due to the nature of how the control using the keys work, when you release the keys, it doesn't show the decay of the music. This will allow a visual element which will show the decay.

Two rendered spheres. The top is controlled by amplitude of synthesiser, bottom is controlled by loaded in track.

Making the Visualiser Spin

Making the Visualiser Spin was actually a very easy task. The Jitter object that renders the image had an attribute called 'camera'. To move the camera, I created a message 'camera $1. $2. $3.' each input controlled how the camera was positioned. The first control controlled how the camera span around the rendered objects. To make the camera spin on it's own and then spin back, I used a metro object to output regualr bangs which was fed into a counter which counted the numbers going up and down. To make the numbers more realistic though I had to divide them by 100 or the camera span too much. Combining these objects allowed the camera to spin at a slow enough pace that didn't distract from the visuals but gave the visualiser another depth. I felt it now related more to Carsten Nicolai's unicolor installation as it gave my project a much bigger depth.

Screenshot of micro patch causing camera to spin. It's also loaded
with a loadbang object so it begins to spin from patch opening.

Controlling Visualiser with Keys and Amplitude

The last development post showed the visualiser being controlled with the keys on the Microkorg. However, the velocity made the 3D bars static and boring. The next step was therefore to find a way to route the audio of the synthesiser to each rendered bar.

One of the rendered bars. Main elements of discussion is circled

The objects that are circled in the image above are what allows the bars to be controlled by amplitude. The 'r k1' object is the receiving the velocity of the corresponding key being pressed. The 'r band1' is receiving the amplitude. These are both being sent to a multiplication object which is then being fed to control the scale of the 3D bar. This works because even if the amplitude being sent is high, unless the key is being pressed sending a velocity number, the amplitude is being multiplied by 0, so it's equal to 0 meaning no scale increase. However, what I found was because the velocity went up to 15, as it was being clipped at 15, the scale got pretty big when being multiplied by 15. To fix this, I changed the pong object so it was now being limited to 1. This kept the scale under control and looking good.

Screenshot showing that velocity is limited to 1 with pong object.

The keys were now doing exactly what I wanted this project to do now. Each key controlled a corresponding 3D rendered bar, with the scale being dictated by the amplitude of the audio. It looked brilliant. But I wanted to make it look a bit more interesting. I was inspired by how another project was spinning as it was showing it's visualiser. See blog post Other Interesting Max MSP Visualisers . I also wanted to add another element that would allow a backing track to be represented with a 3D rendered shape. This would be the next step that will hopefully finish the project.

Other interesting Max MSP Visualisers

As my project is entirely based in Max I did a lot of searching around before building my own visualiser. Although mine will never be on the same par as with these projects, these other visualisers did inspire me to what the capabilities were of using Max and Jitter. It was good as an approach to think "if I aim for moon, I'll only get to the clouds, but if I aim for the stars, I'll get to the moon". In relation to the my project, if I aim for the best project, it will turn out much better than if I aim for it to be just as good as it can be. 

Above is a visualiser that uses frequency to create visuals. This was one the ideas that inspired my project to be controlled by frequency too. Although it was very difficult to work out. However, aiming for this project did lead me to learn how to read amplitude of audio, which was one of the fundamentals to moving the scale of my objects using amplitude.

No real features of this visualiser really made it into my project, however this was one of the first visualisers I came across that was controlled by Max. So this showed what could potentially be the maximum potential of what my visualiser could be. I like how fluid this visualiser is to.

This project is more like my project, in the way that it looks and works. But what relates this most to my project is the way it is controlled using hardware. This is using the Toshio Iwai theory of breaking the boundary between physical world and virtual world by using an interface to control the visuals. I like the way this project is always spinning, which is why I want my project to always spin as it keeps the visuals looking interesting rather than static.

This visualiser is one of many made by someone called Synzou Nagai. His visualisers are amazingly complicated and produce some beautiful looking visualisations. What makes his projects amazing is that looking at the MAX patches, they don't look complicated. But once you begin reading into how he uses the objects with recorded videos of lights and objects they get incredibly complicated. These visualisations don't compare that well to my project, as there is no external control, however the way it is controlled with frequencies and amplitudes makes the visuals look spectacular. His websites shows even more of his projects.

http://synzosizer.com/m_m.html

Carsten Nicolai/ Alva Noto

Carsten Nicolai, also known as Alva Noto, creates a lot of Data Visualisation projects. His works span over quite a few subjects, but one of the subjects I found interesting was his unicolor, unidisplay, and unitxt installations. These installations are a series of projects designed to great ever expanding moving images of colours and shapes, some of which were created with audio hardware.

Unicolor Installation

Nicolai creates these ever expanding images using mirrors. When actually viewing the Unicolor installation, it can confuse the viewers perception as the images are always changing and they expand infinitely due to the placement of the mirrors. This is to create an impression of an ever expanding universe. This can relate to my project, as the rendered images are in an ever expanding 3D plane. The way the 3D shapes work in my project, they can go off the screen, particularly when the pitch is bent, to create this same effect of an ever expanding universe.  To try and relate Nicolai's work to mine, I may try and find a way to spin my project continuously, to show how big the 3D space really is, to create this perception of an ever expanding Universe.

A project made by Nicolai actually helped inspire my project, when it was the LED piano, but still relates heavily to my continuous project. This was a live show of Carsten Nicolai working with musician and composer Ryuichi Sakamoto, to create this data visualisation along with a music performance. Sakamoto is on the piano and Nicolai is using a laptop and together they create this ambient music. However, what makes this performance more interesting is the data visualisaion projected behind them. The data visualisaion brings more life to their performance and creates more meaning to the music. What makes the visuals so effective for them is that they are subtle. The visualisation doesn't distract from the performance, it enhances it. This is exactly what my inspiration for this project was, particularly as I consider myself an ambient composer.

References

Nicolai, C. (2015) Unicolor. [Online] Available from: http://www.carstennicolai.de/?c=works&w=unicolor [accessed 21 March 2015]

Robin Chan. (2012) Alva Noto Ryuichi Sakamoto Live in 2012. [Online] Available from: https://www.youtube.com/watch?v=8ggEYoVKX_M [accessed 13 November 2014]

Control with keys

The next step was to try and get the individual keys on the Microkorg to control each cube in the visualiser. In my mind I knew how this could be achieved, but actually tackling it was going to be difficult.

I began by experimenting with the MIDI related objects in Max, loading in different objects and using the help files to explain what each did. I soon discovered that the midiparse object will print everything that is happening to a MIDI controller. This included: note being pressed, velocity, dials being turned, how much the dials were being turned and more. What made it interesting for me was what it could offer my project. I noticed, if I played a chord, for example C, E, G, even if I played those notes at the exact same time, it would produce the messages for each key, including velocity. Plus, when I removed my fingers, it would send messages to show that the keys were not being played, as it would send out a 0 velocity for each key. This lead me to the idea of having separate micro patches that I could somehow teach to only send out a message, when the corresponding key was pressed, to change a corresponding 3D bar.

Above is a screenshot showing part of my prototype patch that allowed individual key control. Each object send is being sent to an unpack object which allows the MIDI signal to be recognised. I had 37 send objects and 37 recieve objects, 1 for every key on the Microkorg. The next part was very difficult. I knew, from coding in the past, that I could use an "if" statement to filter out the other MIDI key messages to leave only the MIDI key I wanted. After much trial and error, and much help from the max help files, I figured out the "if" statement. Below is an example code that allows MIDI key 48 to filter out so that only, if the MIDI key 48 is pressed, it will allow through.

if $f1 == 48. then out2 $f2. else

Basically what this means is: if the floating number coming into port 1 is received and it is equal to 48, then send out of port 2, the velocity. If the signal coming into port 1 isn't equal to 48, then do nothing.

After I wrote an "if" object for every MIDI key, I moved this patch to my project patch and sent each one out to their corresponding 3D rendered bar. I then routed them so that the velocity would control the scale of their corresponding bar. However, what I found was that because the velocity could be anything between 1 and 127, the scale was so big it would go off the screen. To limit the signal, I used a pong object to limit velocity to 15. Again however, this meant that normally the velocity was 15, making the 3D bars static and boring. Therefore, my next step of development was going to be finding a way to use the amplitude of the audio of the synthesiser, to control each 3D bar. Below is a screenshot  showing part of the patch with the pong objects and the recieve signals controlling the scale of the 3D objects.

Toshio Iwai - Main Inspiration and Practioner

Toshio Iwai is an installation artist who focuses on creating interactive installations, mainly relating to sound and visual. One of his best known installations is the 'Piano - As Image Media' project. This project used a trackball to create visual bars of light that was in sync with sound. This was the primary inspiration of my project, as the theory of Iwai's work is breaking the boundary between the physical world and the virtual world through data visualisation. My project uses this same theory by using the physical world, the synthesizer, to control the virtual world, the visualiser. I also used the same idea of the horizontal streaks of light that represent the keys being pressed from Iwai's installation; in my project I have horizontal beams that represent the individual keys of the synthesizer.

Piano - As Image Media Installation. 

Another work by Iwai that uses a similar theory, of breaking the boundary between the physical world and the virtual world, is his Composition on the Table installation. This installation uses projected images on a table that can be controlled with your hands to create music and sounds. In some ways it actually relates more to my project as it has different elements to control sound, such as dials and switches.  Sounds are produced in relation to the way the projected images are moved. Essentially my project is a mirror image of this being that his project uses controlling the virtual images to create sounds, my project uses sound to control virtual images.

Composition on the Table installation.

References

Young, D. (2010) TOSHIO IWAI. [Online] Available from: http://www.inventinginteractive.com/2010/02/05/toshio-iwai/ [accessed 13 November 2014]

technOasis. (2000) Toshio Iwai. [Online] Available from: http://www.siggraph.org/artdesign/gallery/S99/big/artists/Toshio_Iwai.html [accessed 19 March 2015]

New Proposal

New Proposal - Focus on 3D visualiser

Proposal Project - MUS303 Interactive Sound Environments

Areas to be explored

This work will be exploring the ideas of data visualization, in particular with MIDI note information and audio. The MIDI information is going to be received from an synthesier and/or MIDI keyboard, and audio from either a music track or from the synthesiser/instrument. This may explore ideas around Toshio Iwai, with his “Piano - As Media Image” (Young, 2010). It will also look at performance based visualisation the same as Alva Noto and Ryuichi Sakamoto (Robin Chan, 2012).

Description of Project

For this project, I am proposing that I build a 3D visualiser that can be controlled with the MIDI out of a hardware synthesiser (MicroKorg). Elements that will also control the synthesiser will be the audio from the synthesiser and any backing tracks played at the same time. The visualiser will be mainly based on 3D shapes with each individual shape being controlled by a specific key on the synthesiser. Dials on the synthesiser, such as cutoff frequency, attack, and pitch bend, will control elements of the visualiser, maybe turning the shape or increasing the size. All the shapes will be of multiple colours.

What will this achieve?

When going to see live bands with keyboardists, they are often hidden behind keyboards, with little ability to express their skills with a visual indicator. This differs, for example, from a guitar player. A guitar player has his/her instrument facing the audience, and he/she can move about that stage freely to express the skills he/she possess. That being said, there is an instrument which can be used by keyboardists to reflect this very problem. A keytar. However, I found when watching live keytar players, that they are limited again to what they can do. Keytars are best played with one hand, which limits the skill of the player.

What the visualiser will achieve, is a visual indicator of the skill and expression of the keyboardist. Not only that, audiences enjoy live visuals during performances, and having shapes changing and moving with the music in time with the keyboardist can only increase the enjoyment of the show. Much the same way Alva Noto and Ryuichi Sakamoto do during their live performances.

So the aims of this project are:

• To create a 3D visualiser using Max MSP
• The visualiser should be controlled by a MIDI keyboard and/or hardware synthesiser
• Besides the MIDI information controlling the visualiser, the live sound coming from the synthesiser and/or a backing track should be able to control it.

Breakdown of how to fulfill this task

1. The first task will be to create a 3D visualiser that can be controlled by a prerecorded track. The visualiser should have multiple 3D shapes of multiple colours that pulse or change shape to the music.
2. Second task will be controlling the visualiser with dials on the synthesiser or MIDI keyboard. This will break the boundary of using the physical world to control the virtual world. The dials should be able to rotate the shapes or change the size/scale of the shapes.
3. Third, the shapes should be controlled by the individual keys on the synthesiser. So by pressing a key, it should change a specific shape by increasing the size of the shape. The size should be controlled by the velocity of the key pressed. Also, what would be good at this point, would be to have some element of having shapes controlled by a backing track at the same time of being controlled by the keys of a synthesiser.
4. The last task will be getting the shape's sizes to be controlled by the audio of the synthesiser, but also the key being pressed determines which shape is being controlled by the audio. It would be good to have the shapes move around the camera or the camera move around the shapes to give the visualiser different perspectives of the shapes.

Issues that I may come across

The biggest issue I will need to overcome will be getting individual shapes to be controlled by the individual keys on the keyboard. This is because there will need to a shape rendered for every key on the Microkorg and I will need to find a way to get Max to understand that I only want the specific key to change the shape. 

After being able to control shapes with velocity, it will be tricky to find a way to control the shapes with the the audio of the synthesiser and still have the keys mapped to the specific shapes. This will require multiple micro-patches and it's likely the Max Patch will get very big.

Being able to have a backing track as well as an audio input from the Microkorg could also prove to be difficult as it may be hard to seperate the two audio tracks from each other.

Bibliography

Robin Chan. (2012) Alva Noto Ryuichi Sakamoto Live in 2012. [Online] Available from: https://www.youtube.com/watch?v=8ggEYoVKX_M [accessed 13 November 2014]

Young, D. (2010) TOSHIO IWAI. [Online] Available from: http://www.inventinginteractive.com/2010/02/05/toshio-iwai/ [accessed 13 November 2014]

Change in Project

As it was becoming more difficult to work on both the 3D visualiser and the LED keyboard at the same time, I have decided to drop the the LED keyboard project and put all my focus into the visualiser. I chose drop that project because to create my visualiser, I am using Max MSP and Jitter, which is something that I already have a good understanding of and, being that it is mainly computer based, it would be easier to work with. The LED keyboard however, because I was using an Arduino, there was a big learning curve to understand the code in that. Not only the code, the electronics side was another learning curve. Plus, being able to have 24/7 access to an Arduino was difficult to have unless I purchased one for myself, which I did not have the financial capability to do.

I also worried that my LED keyboard might not work or that it wouldn't be finished in time for the assignment deadline. Furthermore, I was also aware that the LED keyboard on it's own was not to a large enough scale that would be interesting to a live audience; so if I was to drop one project or the other, it was always going to be the LED keyboard, regardless of how much of a learning curve there was. I do have a desire to finish this project, but it will have to be in my own time. 

As a result of this new focus, I have written up a new project proposal that focuses on the visualiser. This will be in a post after this one. 

Visualiser Progress, and using MicroKorg

The past week has been entirely based on experimentation, and building on my Visualiser. The visualiser is coming along nicely.

The progress to lead to this was a number of "prototype" patches. At first I tried to create a visualiser that could be controlled by individual frequencies in a song. But through much trail and error (and a lot of research) it soon became apparent that building such a patch would be bigger than what I hoped my whole project patch to be. However, from doing the research and trials and errors, I built a patch that meant multiple shapes could be controlled with the amplitude of a song or audio input.

Below is a short video showing the prototype visualiser. What this video shows is the attempts of trying to create the frequency controlled visualiser (you may notice the sub patch which was the main body of that trial). Although this didn't work, it does show that I have successfully created a visualiser that uses the amplitude of a track to control the upwards scale of the cubes.

This was the result of 8 different prototype patches. In the last 2 days I started a whole new patch and copied over the parts I liked from the various prototypes I had all ready created. This new patch is laid out a lot more logically. What this new patch starts to incorporate as well, is it is now able to be controlled by a MIDI keyboard.

This would also be a good opportunity to announce that my final project will most likely be controlled with my MicroKorg, and not a USB MIDI keyboard. This allows a much more fluid control over the sound/visual as it will be controlled within the one keyboard, rather than routing a DAW or Plug-in into MAX.

The new patch so far can be controlled and altered in a three ways. First, it can be controlled by the amplitude of a track or audio input (input can be anything, but in this case the MicroKorg). Second, the various cubes (37 in total) can be rotated with the dials on the MicroKorg. Third, the width of the cubes can be increased by using the pitch bend wheel. Below is a video presenting these features.

The next aim is that each cube can be controlled with the individual keys on the MicroKorg, and velocity will control how big the scale is for each cube.

Below are some screenshots of the patch in it's current form.

Another Element

The LED project will be good for audiences who are close to the performer. But anyone else who is further back might not be able to appreciate this. So I have decided to try and include another element, a Visualiser.

To do this, I will be using Max MSP using mainly Jitter objects. With Max/Jitter I will be able to create a 3D shaped visualiser that can be controlled using the same MIDI information that controls the LEDs. This should mean that the LEDs on the keyboard will be in sync with the visualiser.

It would also be a nice feature to have, not only a visual based element of the keyboard, but also a visualiser so that if a backing tack is being played, that track will be represented.