Nissan Music Mixer

UPDATE: This project won a DMA Gold Award (whatever that's worth)
http://www.dmaawards.org.uk/content/2011gold-best-use-social-media-or-viral












I recently worked on a really interesting project for Nissan, which essentially was to be a live mixing desk. As doing anything 'clever' with sound in Flash is something most people (bar Andre Michelle) steer clear of I was looking forward to seeing how much things had changed since my last sound exploits. Not being a maestro of digital sound/music creation, I knew this was going to be fairly tough assignment but one I was going to learn a lot from.

The brief went something like this. The application would consist of five sound channels each channel being a different component of a song (e.g. drums, bass etc). The channels would be represented by cars (the Nissan angle). Each channel would play one of five loops, each loop being a different genre of music (e.g. funk, ambient, rock) which could be changed by clicking on the car. It was also to be possible to change the volume of each channel by moving the car forward and backward. The idea then was that the user could mix and match genre on the fly, bringing in and out different components of the track to create their own master piece. There needed to be a number of sound effects that could be applied too, such as envelopes, flanges and such like. Once they had a play around, they could then record a mix, which would be uploaded to a server as an MP3. Oh, and they wanted some lighting effects in the background (such as fireworks) and a spectrum analyser! Nothing too much then.

Initially there were a number of technical questions I had, like what the hell is a flange and whether it was even possible to create an MP3 from within Flash! Also, how much control we had over the loops (which were to be made by LaRoux) in regards to length and BPM as I had an idea matching up beats was going to be very difficult, given Flash's slack regard to accurate timing!

So, beat matching... firstly it was decided that the loops should all be the same length and the same BPM as this would make life easier. Naturally there were assets available but luckily one of the guys I was working with was really into his music and knocked up some beats for me. It soon became apparent that Flash's onSoundComplete event is hopelessly inefficient for beat matching as there would always end up being a very small delay between stopping and starting a sound, which was never going to acceptable.

Eventually (with a little help from my very talented friend Kelvin Luck!) I turned to ByteArrays. The solution was to start a single 'empty' sound and add a SampleDataEvent.SAMPLE_DATA event, which would be called everytime the sound buffer was empty. It was then a case of reading the correct samples of each track that was designated to play, alter the volumes of the samples to that which were set, add any sound effect that was on at the time and then write this out to the event data to be outputed as one sound. By doing the mixing in Flash and playing a single sound, things would never get out of synch, which was always the problem playing and changing multiple sounds at the same time.

So first I needed to start the empty sound and listen for the SampleDataEvent.

_sound = new Sound();
_sound.addEventListener(SampleDataEvent.SAMPLE_DATA, onSampleData);
_sound.play();

I would store an index for the current sample to read which would be incremented by 4096 (1 second) each time the buffer was empty. When the function was called, I extracted the appropriate samples from each of the tracks that were playing.


//Get index
var n:int = _currentSample + 1 *SAMPLES_PER_EVENT; //SAMPLES_PER_EVENT was set at 4096

//create Vectors for the extracted audio and the volumes
var extractedAudios:Vector. = new Vector.();
var volumes : Vector. = new Vector.();

//and another Vector for the final output
var buffer : Vector.> = new Vector.>();
buffer[0] = new Vector.();
buffer[1] = new Vector.();

for each(var loopId:int in loopIds) {
var playableLoop:PlayableLoop = loops[loopId];

if (playableLoop.isPlaying) {
//store the volume of the track
volumes.push(playableLoop.volume);
//create a new byte array
var sd:ByteArray = new ByteArray();
//get all the samples up to the total samples of the loop
var samplesLoaded:int = playableLoop.sound.extract(sd, SAMPLES_PER_EVENT, _currentSample % _totalSamples);
//get some more samples from the start of the track if the loop got to the end
if (samplesLoaded < SAMPLES_PER_EVENT) {
playableLoop.sound.extract(sd, SAMPLES_PER_EVENT - samplesLoaded, 0);
}
//reset the byte array position
sd.position = 0;
//add the byteArray to the Vector
extractedAudios.push(sd);
}
}

Then I looped through all the extracted samples multiplying the bytes by the volume of the channel and then looped through again, adding the bytes together.


for (; _currentSample < n ; _currentSample++) {
var leftChannel:Number = 0;
var rightChannel:Number = 0;

var index:Number = 0;

for each(var soundData:ByteArray in extractedAudios) {
leftChannel += (soundData.readFloat()*volumes[index]);
rightChannel += (soundData.readFloat()*volumes[index]);
index++;
}
buffer[0].push(leftChannel);
buffer[1].push(rightChannel);
}

This could perhaps be done in the first loop to make it more effecient, but it worked fine so I just left it as such. I needed to add the effects too. I don't have the knowledge required to write sound modulators from scratch so adapted some I found on popforge by Andre Michelle. These took a Number Vector as input and made the mathematical changes for the required effect.

With all that done, I could loop through and write out the byte arrays into the event.


for ( ; i < len ; i++) {
event.data.writeFloat(buffer[0][i]);
event.data.writeFloat(buffer[1][i]);
}

Eh voila, the sound mixing part of the project was done!

The next challenge was to create an MP3 from within Flash. Again, a tricky task indeed, if infact it could be done. After a whole lot of googling and experimenting, it seemed that it was going to be possible to create a WAV from within Flash, and then post that to the backend to be converted into an MP3. I think someone 'may' have written a WAV to MP3 converter in Flash using Alchemy (I vaugely remember seeing something like that), but the client was happy enough with a WAV, which in itself was pretty tricky to do.

Luckily I found a WAVWriter class by Bernard Visscher, which took the samples as a byteArray, adds a header and rewrites the samples so that they are read correctly when played as a WAV. Job done! To test I saved the WAV to the desktop using the FileReference class and it worked great, other than that Flash can't time 30 seconds accurately and the tracks were usually about 36 seconds long using getTimer() in Flash to time 30 seconds! I soon found John Dalziel's Timekeeper class which sorted this out without any dramas.