Instant Musician

iOS AR Piano App

My Role

UI / UX Designer


Jun.2018 - Current

iOS store


Instant Musician is an iOS app that guides people through playing music on the piano. It leverages ARKit, and is a sister product to our award-winning HoloLens app, Music Everywhere. This product is the second AR music app from my startup Music Everywhere, Inc which consists of one programmer, one musician and one designer - me.

After our success on the HoloLens platform with Music Everywhere, we connected with many other AR hardware manufacturers in the hope that we’d land on a platform that could reach a larger user base; in other words, something that was cheaper than the enterprise-focused HoloLens. These companies included Magic Leap, Meta and Lenovo’s AR headset (launched with the lightsaber controller for Jedi Challenges).

Unfortunately, these products had tracking and visibility limitations that convinced us that AR head-mounted displays are not quite ready for the consumer market. One viable platform, however, remained: Apple’s iPhone.


Stage 1 - What problem will we solve with Instant Musician?

Problem and Opportunity

Learning how to play an instrument is really hard. You need a good teacher, you need to practice a lot, and you often need to spend a significant amount of money before any of that. And you have to stay committed to learning this instrument for a long time before you can start playing the songs you enjoy.

learning curve

While playtesting Music Everywhere with the HoloLens, we observed a funny thing:
People didn’t want to learn, they just wanted to play.


Using Augmented Reality (AR), we let people bypass this tough music learning curve, and skip to fun part - playing songs!


End User

My team want to reach users who feel left behind from learning music due to limitations in conventional music education. Our AR solution opens the door for people to play music in the moment, regardless of musical ability.

Competitive Landscape

Competitive landscape

Stage 2 - Integrating mobile AR with the piano

Challenge 1- Holding the phone while playing

It’s awkward playing the piano with one hand, especially while holding an iPhone at the same time.

Challenge 2 - Lowering the product’s friction

My team’s primary goal for this app was to make piano playing easy for our users, regardless of their music knowledge. In some instances, this meant eliminating the obstacles that keep people playing the piano in the first place. In others, it meant eliminating the friction that came with an AR interface.

Design Solutions

Making it comfortable to hold a phone

I know, holding a phone can get tiring (the thought of holding an iPhone XS Max for an hour makes my thumbs cramp!). To help mitigate this, My team made the songs shorter (00:30-1:00 in length), and I organized the songs in skill level order to give users a sense of accomplishment with each completed song. This broke up the experience into cycles that helped offset any fatigue.

Finding a universal marker

To ensure stable alignment, image tracking markers are used in many AR experiences. From developing the HoloLens experience, I knew I wanted to avoid a custom marker (more friction for the user) and focus on something the user already had. Using a $1.00 bill allowed us to use a marker that was readily available for most users.

Stage 3 - Designing the user interface

Challenge 1 - Identifying the player function

The app’s core function is guiding users to play songs with AR musical notations. To achieve that, we needed to consider how people interact with a song’s background music.

Challenge 2 - laying out UI elements


Big AR camera screen

The most important AR design principle is making the AR camera feed as big as possible on the screen. A well-arranged UI layout is critical to achieve this goal.

Left hand friendly

Users will hold the phone with their left hand, and play songs with right hand. So, all the on-screen interactions are better to be “left hand friendly” to allow users to select options in an efficient way.

Challenge 3 - Helping users see every note

The visibility of the musical notes is critical to the experience. What happens if it’s out of frame on the phone screen? How do users know where to look on the keyboard first?

Design Solutions

Designing Play Mode / Practice Mode

In play mode, users can adjust the background music like its speed and accompaniment styles. For active users, we design the practice mode for them to rehearse the song, one musical phrase at a time.

play mode
practice mode

Making the “UI Drawer” system

In order to create a clean and big space for the AR camera feed, I designed an UI drawer system to store UI elements, like switching between play / practice mode, changing speed, etc.

UI drawer Interaction

UI drawer Interaction

UI drawer closes

Drawer closes

UI drawer opens

Drawer opens

Designing the note location indicators

1. A white boundary on the sides of the screen will appear when there are notes outside the screen’s view to guide users to move the phone in that direction to see them.

White Boundary- Find the notes outside the view

Find the notes outside the view

Left boundary

Left boundary

Right boundary

Right boundary

2. An interactive compass will help users to find the start point after they pick a song. Instead of playing immediately, in this way, they have more time to get prepared.

Interactive start point finder

Interactive start point finder

Campss finder

Campass finder

Start point

Start point

Stage 4 - Guiding users how to play

Challenge 1 - Onboarding users

iOS AR is still new to the majority. Therefore, the onboarding session plays an important role to give users an overview of how the AR technology work in this app at beginning.

Challenge 2 - Guiding how to register the AR interface

Like our HoloLens product, we needed to teach users where to put the image tracking marker (a dollar bill in this case) to spawn the AR interface, and adjust its alignment as well.

Challenge 3 - Guiding how to play

We needed to have a tutorial to guide users through the important UI functions.

Design Solutions

Creating functional onboarding pages

I combined the welcome messages and the AR setting overview to create an appealing and functional onboarding session.

onbaord 1
onbaord 2
onbaord 3
onbaord 4
onbaord 5

Making an interactive marker alignment guidance

I combined the welcome messages and the AR setting overview to create an appealing and functional onboarding session.

Find Middle C
Align the dollar bill
Register AR UI

Creating step-by-step play instructions

I illustrated a sequence of graphics to cover core functions.

play 1
play 2
play 3
play 4

Stage 5 - Designing the Store Page - Songbook

Challenge - What’s the business model?

The in-app purchase flow tights with the business model. How we monetize the app determines the design for the store spage.

Design Solution

Buying complete genres for songs

The licensing for an individual song can change. For example, songs on Spotify or iTunse grayed out after the licensing transferred to others. We decided to make a genre package includes 5~8 free songs, and 15+ locked songs. Users can buy the complete genre for all songs. In this way, we can replace the grayed out songs to protect their purchases.

Songbook interaction

Songbook Interaction

Songbook button

Songbook button

Song book IAP

Buying complete genres for songs

Stage 6 - Materials to give the app a brand identity

Below are the materials I created that represented our app’s design and experience.


Stage 7 - Test & Iteration

A usability testing was conducted with a beta version of Instant Musician using Apple’s TestFlight. The test was designed to assess the success of the AR play function; its efficiency and usability when performing the designed tasks flows; The overall test findings identify errors and gain user’s feedback.

Future Design Improvements

Play tutorial session

A good tutorial experience will guide users to learn by doing instead of providing still images. To do so, we can design a simple song with tutorial contents for users to learn interface functions while playing.

Access to the songbook

The songbook should be always accessible. No AR interface registration required.

Looking Ahead

Creating a consumer-focused AR piano playing experience that utilizes both hands at once is the ultimate goal. Our eyes are on Apple’s coming consumer-ready AR glasses (2021), and adapt Instant Musician to this exciting new platform, converting iOS users from our current app.

Road map

AR piano learning road map