My Role

I led the design and 3D art of Music Everywhere on the Microsoft HoloLens since the project’s inception in Sept 2016. In Nov 2017, I led the team’s global launch of the product on the Microsoft App store.
We won the 1st place in the Microsoft HoloLens Contest. Along with the $100K grand prize.

1st place

To read more about my experience with the Microsoft HoloLens Competition, please visit my design blog.

Context

Music Everywhere started as a graduate school project during my time at Carnegie Mellon University. Our team consisted of two programmers, one musician and one designer—me. Our goal: build an Augmented Reality (AR) piano learning tool.

The Problem

Hard to Find Which Key to Play From Instructions

Most piano learning apps are designed for mobile phones and tablets. A major issue for users during play comes when users must mentally translate the instructions from the screen to the piano keys.

Our Solution

Augmented Reality

With AR technology, we align the 3D interface (the piano instructions) directly on the physical piano keys. This makes discerning our AR instructions with the actual instrument easier and more intuitive.
To read more about the AR Interface Design, please see this NIME2017 paper we published.

solution

The Discovery

Target Customer

One segment of our userbase included music teachers who wanted to explore new ways of reaching students in music education. Below is an example of a persona that reflects this customer.

persona

Customer Insights

In order to create viable features, we interviewed music educators and pianists in Pittsburgh & New York.

  • Improvisation

    Often missing from traditional curricula, musical improvisation is important because it enhances performance skills, develops one's musical ear and improves understanding of theory and context.

  • Virtual Hands

    The Virtual Hand would be a good tool to demonstrate hand positioning and performance techniques in the AR space.

  • Virtual Musicians

    3D virtual musicians provide an optimal environment to learn about and practice improvisation concepts, scales and techniques.

Design Process

How We Got There

Envisioning the experience

I previsualized the interactive piano lessons, where users can follow the play guidance from virtual musicians.

sketch-1

sketch-1

Developing the AR 3D piano interface

1 - Design problem

HoloLens’ FOV (feild of view) causes users to have to angle their head downward in order to see the cut-off keyboard.

AR UI Dev FOV

2 - Design approach

  • First, start with overlays on keys, then cut the overlay to half to quarter in order to decrease the dependency of the overlay gradually.
  • Second, in order to guide users to look up a little bit, we remove the overlay on keys and put markers which point the key to play.
  • Last, I designed floating keys hovering in the air and a mirror keyboard in order to encourage users to look up.
AR UI Dev FOV

​3 - Note Overlay Development

Note Overlay Dev

Accurate Shape
No expectation for the next note

Note Overlay Dev

Charging Bar
Show a sequence of note for one key?

Note Overlay Dev

Layer Bar
​Too many colors

Note Overlay Dev

Sequence​-Final Version
Clean visual/Show a sequence/expectation of note

Making the setup experience seamless

In order to ensure a smooth setup, we needed account for:

  • 1. Connecting the HoloLens with the digital piano via Bluetooth.
  • 2. The user’s digital piano size.
  • 3. Generate, align, and lock the virtual keyboard on the real piano.
  • 4. Solve a Bluetooth disconnection.
  • 5. Re-align the piano when it drifts away.

1 - Integrating technology - Bluetooth

In order to determine which note users played on the keyboard, we used a Bluetooth connection to send out a MIDI signal from the digital piano to the HoloLens. This made ensuring a good connection critical to the experience.

bluetooth

We used a Bluetooth connection to send out a MIDI signal from the digital piano to the HoloLens.

Due to the native HoloLens environment, users must pair Bluetooth devices through the device’s system settings. To help guide users, I created an animation to teach them how to complete this process, and included a tutorial for the Bluetooth connection process troubleshooting as well, in case the original setup was not done correctly.

bluetooth connection

2 - Account for different piano keyboard sizes

We cover 88, 61 , 49 and 25 keys of size of pianos, which accounts for almost every kind. The system can detect the piano size after users press the first and the last key, and adapts the AR interface to the keys.

keyboard size

3 - Guiding users to setup keyboard alignment

For a shipped product, we knew we needed to create an intuitive process for users to do this themselves, successfully and reliably.

A customized image tracking marker is within hand’s reach for the user to spawn the AR interface. By looking at the tracking marker, users can spawn the interface.

Then place it on the middle C to adjust the alignment by easily moving the markers around.

-- Finding Middle C

find Middle C

-- Aligning the Marker on Middle C

align marker

Making engaging lessons

lessonflow

Our goal is to guide and inspire people on how they can express themselves with playing piano, through improvisation. In our lessons, the virtual musicians teach improv knowledge. Users can follow along with instructions in Following Mode, or experiment with different notes in Improv Mode.

1 - Designing a scalable lesson structure

We designed the lesson packages which included several lessons within them. This allowed us to design a system that accounted for every music level, to define each lesson package’s level, and to upload more materials for the future.

lesson package

Pick a lesson package accoring to music level

lessons

Pick a lesson inside the package

2 - Building the troubleshoot function

AR experiences often tend to be unstable due to technical limitations today. For example, the AR interface might drift away from the current alignment or the Bluetooth connection might disappear. We knew it was critical to craft troubleshooting functions for users within the app to build our next version.

To reach the goal, I designed the torubleshoot process. When problems happen, users can pause the experience in place to resolve the issue.

troubleshooting

3 - AR character development & energizing the experience

People love animated characters. They’re fun, liven up a scene, and provide a fun way to instill the context of music playing with a band. To achieve this in a meaningful way, I built up animations to sync with music beats and created more interaction scenarios.

talk

A Virtual musician step towards the user to teach them how to play

play

Animations sync with music beats

Old virtual bands

Virtual band prototype

Validation

Testing Our Designs

tester

The tester is testing the HoloLens and piano connection

To design for the HoloLens UX, being pessimistic is a good attitude.

“What if our design totally fails, or for people who don’t have the Bluetooth compatible digital piano?”
Considering the worst cases, I design the “No-Piano Demo”option for people who can’t get the settings to work, or without a proper piano. Through this demo, people can get a preliminary understanding for how Music Everywhere works. From this pessimistic perspective, we developed a full FAQ section. Here, users could troubleshoot by themselves with guided instructions, and get to know more details about Music Everywhere.

FAQ

FAQ: Supported Piano, No-Piano Demo

Visual Elements

Below are the Visual assets I created that represented our app’s design and experience.

branding

Recognition

Conference

conference

Articles & Reviews

Funding Partners

Looking Ahead

Working on the Microsoft HoloLens platform allowed us to be an early publisher of AR music learning content. We got early recognition from industry leaders, but knew that our future would depend on Mobile AR. The HoloLens, though an established platform, is too expensive to ever achieve widespread adoption ($3K USD).

Looking ahead, we knew we would make our next experience using ARKit for iOS, bringing our user base pool from early adopter AR enthusiasts to the population at large. Hello, iPhone.