This recent CodePen Chicago meetup had a Halloween theme. I had an idea to dive into Firebase and create a multi-device musical performance using Tone.js and meetup participants' cell phones. Since this isn't something that you can really convey from sharing a pen, I am describing it here and hope to show just how simple it can be to do something like this on your own.

The Pen

A quick overview of the functionality

Meetup Footage Courtesy @poopsplat

This pen has three functional aspects:

A Firebase real-time database to sync an experience across multiple devices
Web Audio API via Tone.js to play sounds on devices
DeviceOrientationEvent to change sounds with physical device movement.

All of these things can be combined together on CodePen.

Using Firebase, we can modify data on one device and listen for that event on other devices. When that event occurs, we can take the new data on the listening devices and do things with it. This is not unlike listening for a click event on a webpage and responding to that input.

In the case of my Halloween Musical App, my personal computer would update data on my Firebase database and all the devices listening for that Firebase event would play different sounds.

Firebase

Setup

For the unacquainted, Firebase provides a host of services, but in our case we only want to use the real time database. The best way to get that going is to plow through the Firebase "Get Started on the Web" documentation. On the free tier, you can have up to 50 concurrent sessions real-time reading your database.

You will need to include three scripts from Firebase in your JavaScript settings on CodePen:

The list of scripts for copy/paste:

//www.gstatic.com/firebasejs/3.5.0/firebase.js
//www.gstatic.com/firebasejs/3.5.0/firebase-auth.js
//www.gstatic.com/firebasejs/3.5.0/firebase-database.js

Authorization

Once you set up your app and have it authorized, you will need to create anonymous sessions for users to read/write data. The documentation for Anonymous Authentication is pretty thorough.

Usage

After you have authorized your app to read and write from Firebase, modifying and listening for data is pretty straightforward.

To write to your database, it is this easy:

  firebase.database().ref('something').set('REAL')

At that point, in your Firebase console, you would see something like this:

So now you are writing to Firebase from your app. The amazing thing is how easy it is to also have your app live-update with database changes. Whenever we set the data, we can do things like console.log the value:

  // when the firebase value is updated, do something with it
firebase.database().ref('something').on('value', (snapshot) => { 
  console.log(snapshot.val()) // this value is coming from Firebase, not our app
})

// set a string value on firebase
firebase.database().ref('something').set('Hello World') 
// console => "Hello World!"

// wait one second and set it again, but this time set it as an object
setTimeout(() => {
  firebase.database().ref('something').set({ Hello: 'World!' }) 
  // console => { Hello: 'World!' }
}, 1000)

This might seem super basic and boring until you realize that any browser that is listening to your database's something property would have their browser's console log the value as well. In other words, anybody looking at your pen would log these values when you set them.

Tone.js

I have been creating a lot of audio pens recently, and Tone.js has made that process simple. In this case, I wanted to have each person looking at my pen play their own note and harmonize with the rest of the room. Most simply, harmony is when different notes are played together and ideally sound good together. How do you get people running the same code to play different notes? I elected to use randomness and it worked out well enough.

Generating Notes to play

Let's say we want to be able to play three chords. For the sake of simplicity, we will play triads which are chords of three notes. In Tone.js, we can write notes as the letter value and the octave. Middle C (middle key on a piano) would be notated as 'C4'. The note C in the 4th octave. To save energy, I like to use Array.map and Array.split to create arrays like this. It allows me to easily edit or add chords without having to write a bunch of stuff.

  var octave = 4
var chords = [
  'ACE', // A minor
  'DFA', // D minor
  'FAC'  // F major
].map((chord) => {
  // make 'ACE' => ['A','C','E']
  chord = chord.split('')
  // make ['A','C','E'] => ['A4','C4','E4']
  return chord.map((c) => { return c + octave })
})

// chords now looks like this
[
  ['A4','C4','E4'],
  ['D4','F4','A4'],
  ['F4','A4','C4']
]

Now that we have our chords, we want to randomly generate which notes the device will play for each chord. In this case, we want the device to always play the same note when it is asked to play the chord.

  // chords = [
//   ['A4','C4','E4'],
//   ['D4','F4','A4'],
//   ['F4','A4','C4']
// ]
var notes = chords.map((chord) => {
  // randomly get a note from the chord
  return chord[Math.floor(Math.random() * chord.length)]
})
// notes now looks something like this
['A4','F4','C4']

Creating A Synth

To play a note with Tone.js, you need to create an instrument to play it with. First, we need to include Tone in our pen with a link to the minified script //cdnjs.cloudflare.com/ajax/libs/tone/0.8.0/Tone.min.js.

Now we have access to the core Tone library.

To create an instrument, we can just use a default synth.

  var synth = new Tone.Synth()

Well that was easy. Not so fast, in order for the synth to actually make noise, we need to plug it in.

  var synth = new Tone.Synth()
synth.toMaster()

This might seem laborious, but needing to connect the synth to the output is actually very important. This way, we could connect it to other things like delay effects before it gets connected to the output.

Now that we have a synth that is connected, we can play a note.

  synth.triggerAttackRelease('C4', '8n')

This code above is basically saying "play a middle C eighth note". In my app I just wanted to start playing a sound indefinitely. To do that, you simply use triggerAttack.

  synth.triggerAttack('C4')

To stop it we use triggerRelease.

  synth.triggerRelease('C4')

Playing the notes

We are making noise. Now let's create a play function that will play our note based on the chord.

  function playNote(chord) {
  synth.triggerAttack(notes[chord])
}

// our notes from earlier look something like this
// ['A4','F4','C4']
playNote(0) // triggerAttack 'A4'
playNote(1) // triggerAttack 'F4'
playNote(2) // triggerAttack 'C4'

If the synth is already playing a note, it is best practice to use the setNote method instead of triggerAttack. This is necessary for certain effects like portamento (gliding between notes). Our function will change slightly in order to make that work.

  var synth_on = false

function playNote(chord) {
  note = notes[chord]
  if (synth_on) {
    synth.setNote(note)
  } else {
    synth.triggerAttack(note)
    synth_on = true
  }
}

// our notes from earlier look something like this
// ['A4','F4','C4']
playNote(0) // triggerAttack 'A4'
playNote(1) // setNote 'F4'
playNote(2) // setNote 'C4'

For more details on what you can create with tone, see the Tone.js Documentation.

Mobile.

In order to have mobile devices properly recognizing AudioContext and play notes, you need to have a valid user input trigger the instantiation of AudioContext. That's a confusing thing that you don't need to bother understanding, I sure didn't. Thankfully there is a library that can do all of this for us.

First we need to include StartAudioContext in our scripts:

//cdn.rawgit.com/tambien/StartAudioContext/master/StartAudioContext.js

The way this works is you bind it to a button before creating your things. The best way I've found looks something like this:

  <button id="start">Start Audio Context for Mobile!</button>

  var $start = document.querySelector('#start')
// the context will be started on the first valid user action
StartAudioContext(Tone.context, $start, () => { 
  someInitFunction()
  $start.remove()
  console.debug('AUDIO READY') 
})

Since we are using Tone, we pass Tone.context as the first parameter, the button as the second, then the callback removes the button and calls some primary init function to set up our app.

Device Orientation Event

If you have SSL enabled on your website (or are on https://codepen.io), you can access device orientation data. This allows you to access degrees of rotation along the x, y, and z axis and transform that into ratios. I used it to control the volume of each phone's synthesizer. There is a very good overview on the DeviceOrientationEvent docs on MDN.

  window.addEventListener('deviceorientation', (e) => {
  let x = e.beta  // -180deg through 180deg
  let y = e.gamma //  -90deg through  90deg
  let z = e.alpha //    0deg through 360deg
}, true)

So if I wanted to adjust volume for a synth based on x-axis rotation, it would look something like this:

  // Tone.js
let channel = new Tone.Gain(1) // a gain (or volume) node with full volume (0-1)
let synth = new Tone.Synth() // a synth to play notes with
channel.toMaster() // send the gain to the master output
synth.connect(channel) // send the synth to the gain node


// DeviceOrientation
let max_fwd = 80 // max rotation forward (degrees)
let max_bwd = 20 // max rotation backward (degrees)
let max_tot = max_fwd - max_bwd // total rotation possible (degrees)

window.addEventListener('deviceorientation', (e) => {
  let x = e.beta  // -180deg through 180deg
  // enforcing the min and max values
  x = Math.min(max_fwd, Math.max(x, max_bwd)) - max_bwd
  // get a ratio of how much we have rotated
  let rat = 1 - (x / max_tot) * 100 / 100
  // set the channel gain value to that ratio (0-1)
  channel.gain.value = rat
}, true)

Bringing it all Home

This won't be fully operable since we aren't going to do Firebase authorization, but assuming we had authorized our app to read and write, we could build a simple multi-device musical performance with a fairly small amount of code.

The main part of the app that everyone would use would look something like this:

  <button id="start">Let's Ghoul!</button>

  /* 
 * initing the audio context
 */

var $start = document.querySelector('#start')
// the context will be started on the first valid user action
StartAudioContext(Tone.context, $start, () => { 
  init()
  $start.remove()
  console.debug('AUDIO READY') 
})

/* 
 * generating notes randomly
 */

var octave = 4
var chords = [
  'ACE', 'DFA', 'FAC'
].map((chord) => {
  chord = chord.split('')
  return chord.map((c) => { return c + octave })
})
var notes = chords.map((chord) => {
  return chord[Math.floor(Math.random() * chord.length)]
})


/* 
 * init function that relies on audio context
 */

function init() {

  /* 
   * hooking up Tone.js
   */

  let channel = new Tone.Gain(1) // a gain (or volume) node with full volume (0-1)
  let synth = new Tone.Synth() // a synth to play notes with
  channel.toMaster() // send the gain to the master output
  synth.connect(channel) // send the synth to the gain node

  var synth_on = false

  function playNote(chord) {
    note = notes[chord]
    if (synth_on) {
      synth.setNote(note)
    } else {
      synth.triggerAttack(note)
      synth_on = true
    }
  }


  /* 
   * listening to firebase then playing the note for the chord
   */

  firebase.database().ref('chord').on('value', (snapshot) => { 
    playNote(snapshot.val())
  })


  /* 
   * handling DeviceOrientation event
   */

  let max_fwd = 80 // max rotation forward (degrees)
  let max_bwd = 20 // max rotation backward (degrees)
  let max_tot = max_fwd - max_bwd // total rotation possible (degrees)
  window.addEventListener('deviceorientation', (e) => {
    let x = e.beta  // -180deg through 180deg
    // enforcing the min and max values
    x = Math.min(max_fwd, Math.max(x, max_bwd)) - max_bwd
    // get a ratio of how much we have rotated
    let rat = 1 - (x / max_tot) * 100 / 100
    // set the channel gain value to that ratio (0-1)
    channel.gain.value = rat
  }, true)
}

Finally, we would have some logic that allowed us to update a chord value in our database. In my case, I looked for a query param in the url (admin=true) to add functionality to the pen so I could write to the database using my keyboard but no one else could.

  // if session is admin (whatever that means in your scneario)
if(admin) {
  // updating firebase on keydown of 'Q', 'W', or 'E'
  document.body.addEventListener('keydown', (e) => {
    // q:81 w:87 e:69 // letter:keyCode
    let note = { 81: 0, 87: 1, 69: 2 }[e.keyCode] // notes 0, 1, and 2 mapped to keycode
    if(note) firebase.database().ref('chord').set(note); // 0, 1, 2, etc
  })

}

In Conclusion

This was a really fun project and I was really happy it went over so well. I love that both CodePen and CodePenChicago exist as a form of inspiration for all of us. Creativity is the best utility for learning. Keep persevering and making things.