baaack home
Virtual Reality for Data Visualization - Spring 2022 @ Brown University
VR Piano Visualizer
As a piano teacher at the ripe old age of 21 (at least, that's what most of my 7-year-old students think), I hear one thing more often than I thought I would - "I learned the notes by looking at a video on YouTube!"

Who knew the technological generation gap was real, hm?

Either way, I appreciate that these videos motivate my students to learn notes, but how to play the notes is a different question. What's loud, what's soft? That's harder to tell visually.

virtual reality piano application
My final project I created for the VR for Data Viz course I took was a MIDI piano visualizer, in virtual reality. Most 2D piano visualizers on YouTube are able to tell you which notes to press and for how long, but I wanted to take this to VR in order to show how loud the note should be played.

After all, that's what determines whether Clair de Lune will sound like la lune (the moon) or le soleil (the sun)!

Made in Unity.
What makes a good Virtual Reality app?
That's what I had to answer before I could start. So, I used the example of looking into existing VR software that allow users to mask (erase material from) models. Not inherently related to a piano, but exploring this helped me come up with metrics to keep in mind when creating my own application.
evaluation table of modeling software
And so from this, why did I decide to create a piano in virtual reality?
Kids learn in different ways, and one of those is visually. On a normal piano, or even in 2D, you can see what notes to play, but it is harder to distinguish volume just by looking at the notes. Sure, one could differentiate the volumes by ear but not everyone has perfect pitch!
So, an app that's easy to understand, allows multiple people to be in the same room, and has a clear menu system.
That was the goal. Now let's see what I did.
vr piano application's menu
Before you can start the piano... you have to choose a piece!
So first, I had to choose an input. Where is this piano music coming from?

My answer: MIDI files. MIDI files include information on note pitch, volume, duration, modulation, etc. Exactly what I need!

I loaded a few MIDI files to the app for the user to choose from. Of course, with a classic play/pause button and some speed options, all on a simple menu attached to the left joystick - you can't lose it!

pssssst - I used a Unity asset called Maestro Midi Player Tool Kit to translate the MIDI data into something I could use to control the piano model I made. I wrote a tutorial here on how to use it!
Using Normcore's Unity plugin, I was able to bring multiple people in the same room!

vr piano application with other avatars in the room
Looking back, and to the future
Now, despite the features I was able to implement smoothly, there were also a few things that I wish I had spent more time on to debug. For example, I was able to get everyone in the same room using Normcore but, I didn't have time to make synced collaboration! Two people could be in the same room but would not be able to hear the other person's piano. I know, I want this to happen in the future too, since I could see this being a useful tool in my studio!

There are also a handful of other features I would love to see in this app: loading your own MIDI files, notes being different colors based on which hand is playing it, synced sheet music, note letter names, seeing what notes you'll have to play next, etc. While I wasn't able to implement all of these in the month I had for this project, I hope to see them in the future!