My Role

I led the design of Instant Musician on the iPhone with ARKit technology since the project’s inception in Jun. 2018. In Sept 2018, I led the team to launch the product on the U.S Apple App store.

Context

Instant Musician is an iOS app that guides people through playing music on the piano. It leverages ARKit, and is a sister product to our award-winning HoloLens app, Music Everywhere. This product is the second AR music app from my startup Music Everywhere, Inc which consists of one programmer, one musician and one designer - me.

The Challenge

Create a Shortcut to play the piano

Learning how to play an instrument is really hard. You have to stay committed to learning this instrument for a long time before you can start playing the songs you enjoy.

learning curve

The Approach

AR on Your iPhone

Using Augmented Reality with mobile phones, we let people bypass this tough music learning curve, and skip to fun part - playing songs!

shortcut

Design Process

How We Got There

Integrating mobile AR with the piano

1 - Holding the phone while playing

-- Why is Portrait orient?

Although landscape is an ideal layout because of its wider FOV of the piano keys, I discovered that during early testing that landscape mode left users feeling queasy.
Visual Distortion

d might cause people sick. d is bigger in landscape mode

Let's say the horizontal distance between right hand position on the screen and real life is "d".
I found that "d" might cause users sick, and d is bigger in landscape mode.
In portrait mode, users tend to move the phone closer to the right hand to see it on the screen. However, in landscape mode(camera on the right side), the right hand can be seen one the screen even the phone is in the distance.

2 - Designing the AR image tracking marker

-- What is the one dollar bill for?

To ensure stable alignment, image tracking markers are used in many AR experiences. From developing the HoloLens experience, I knew I wanted to avoid a custom marker (more friction for the user) and focus on something the user already had. Using a $1.00 bill allowed us to use a marker that was readily available for most users.

Tracking Marker

Left: Custom marker - More friction for users
Right: A $1.00 bill marker - Available for users

Tracking Marker

Register the AR piano interface with a $1.00 bill

3 - Developing the AR 3D keyboard interface

-- AR 3D keyboard interface

AR keybaord dev

-- AR 3D music notation

AR musical note dev

Designing the phone user interface

1 - Identifying the player function

The app’s core function is guiding users to play songs with AR musical notations. To achieve that, we needed to consider how people interact with a song’s background music.

In play mode, users can adjust the background music like its speed and accompaniment styles. For active users, we design the practice mode for them to rehearse the song, one musical phrase at a time.

features

2 - Laying out UI elements

-- Big AR camera screen

The most important AR design principle is making the AR camera feed as big as possible on the screen. A well-arranged UI layout is critical to achieve this goal.

-- Left hand friendly

Users will hold the phone with their left hand, and play songs with right hand. So, all the on-screen interactions are better to be “left hand friendly” to allow users to select options in an efficient way.

Iteration

Based on the "Big AR camera screen" and "Left hand friendly" layout principles, I started with a basic layout plan. Then I tried to make the BGM/Speed settings and Play/Practice switch button visible on the sreen.

process-1

After I realized I didn't think about big phone users like iPhone 8 and above, I adjusted my layout plan to lower the Songbook button. However, in this way I couln't show other settings at same time, I developed the drawer system to organize them.

process-2

Final UI

I designed the UI drawer system to store UI elements, like switching between play / practice mode, changing speed, etc.

UI drawer Interaction

UI drawer Interaction

UI drawer closes

Drawer closes

UI drawer opens

Drawer opens

play mode

Play Mode UI

practice mode

Practice Mode UI

3 - Helping users see every note

The visibility of the musical notes is critical to the experience. What happens if it’s out of frame on the phone screen? How do users know where to look on the keyboard first?

1. An interactive compass will help users to find the start point after they pick a song. Instead of playing immediately, in this way, they have more time to get prepared.

Interactive start point finder

Interactive start point finder

Campss finder

Compass finder

Start point

Start point

2. A white boundary on the sides of the screen will appear when there are notes outside the screen’s view to guide users to move the phone in that direction to see them.

White Boundary- Find the notes outside the view

Find the notes outside the view

Left boundary

Left boundary

Right boundary

Right boundary

Designing the Store Page - Songbook

1 - What’s the business model?

The in-app purchase (IAP) model determines the songbook design. We started with doing research about how to monetize this app.
The licensing for an individual song can transfer or expire. Therefore, we decided to make a genre package includes 5~8 free songs, and 15+ locked songs. Users can buy the complete genre for all songs. In this way, we can replace expired songs to mitigate the sense of losing purchase.

Business Model

Users can buy the complete genre for all songs. We can replace expired songs with new songs.

2 - UI Development

Songbook Dev
Songbook interaction

Songbook Interaction

Song book IAP

IAP: Before purchase all Jazz Songs

Song book IAP

IAP: After purchase all Jazz Songs

Guiding users how to play

1 - Onboarding users

iOS AR is still new to the majority. Therefore, the onboarding session plays an important role to give users an overview of how the AR technology work in this app at beginning.

I combined the welcome messages and the AR setting overview to create an appealing and functional onboarding session.

onbaord 1
onbaord 2
onbaord 3
onbaord 4
onbaord 5

2 - Guiding how to register the AR interface

Like our HoloLens product, we needed to teach users where to put the image tracking marker (a dollar bill in this case) to spawn the AR interface, and adjust its alignment as well.

Find Middle C
Align the dollar bill
Register AR UI

3 - Guiding how to play

We needed to have a tutorial to guide users through the important UI functions.

I illustrated a sequence of graphics to cover core functions.

play 1
play 2
play 3
play 4

Visual elements

Below are the visual materials I created that represented our app’s design and experience.

branding

Validation

Testing Our Designs

A usability testing was conducted with a beta version of Instant Musician using Apple’s TestFlight. The test was designed to assess the success of the AR play function; its efficiency and usability when performing the designed tasks flows; The overall test findings identify errors and gain user’s feedback.

Future Design Improvements

Play tutorial session

A good tutorial experience will guide users to learn by doing instead of providing still images. To do so, we can design a simple song with tutorial contents for users to learn interface functions while playing.

Access to the songbook

The songbook should be always accessible. No AR interface registration required.

Looking Ahead

Creating a consumer-focused AR piano playing experience that utilizes both hands at once is the ultimate goal. Our eyes are on Apple’s coming consumer-ready AR glasses (2021), and adapt Instant Musician to this exciting new platform, converting iOS users from our current app.

Road map

AR piano learning road map