The Challenge
How to facilitate hands-free music sheet reading for keyboard players?
Team
Design Team of Two; Advance Wearable Course
My Role
Design Research, User Mapping, Circuit Design, Prototype Construction.
Timeline
Feb 2022, 3 Weeks
Tools
Python, Adobe Photoshop, Arduino Uno, Muse Band 2014, relay module, 6.35mm jack, Roland keyboard, phone and laptop.
Project Motivation
As an instrument player, I oftentimes find it difficult multi-tasks through coordinating different bodies parts in a good timing, such as flipping pages while both hands are employed in the act of playing.
Why it matters
In most cases, a page break occurs during a rest for the particular instrument. When it is does not, the musician who is required to turn the page must stop playing. Thus, in a performance, page turning actually results in perceptible "dips" in the music's volume. In rehearsal, page turns may occur in both directions since the conductor may choose to focus on a part of the composition and rehearse it over and over.
Understanding the users
Target User Group
Keyboard/piano players and learners
Journey Mapping
Secure sheets on a music stand
Start playing
Memorize measures before turning pages
Quickly flip sheets with
one hand
Page flipped, both hand return to playing position
Problem Occurred
Interruption
A page flip with a bad timing can easily cause perceivable interruption in a music piece.
Rhythm Disruption
A delayed note causes the rhythm of the whole performance to be compromised.
Mental Stress
Preparing to flip pages at the end measure requires extra mental energy from the musician.
Current Coping Methods
External Help
Hiring a page-turner for performances.
Quick Action
Quickly turn the page when reaching the end of the page.
Memorization
1. Memorize the measures before and after the page turn and play with one hand during the turn.
2. Memorize the whole piece and don’t use the sheet music.
Technology
1. Automated page turner.
2. Bluetooth technology which needs to be controlled by other body parts, e.g. feet.
Market Research
In order to better understand the current market, we looked into products that are either academically or commercially recognized. From our research, products below are three successful page turners that are representative of similar products. We looked at them thoroughly and listed down their advantages and drawbacks.
Pageflip
Pros
-
Hands-free
-
Durable material
-
Wireless connection
-
Affordable ($80 -100)
Cons
-
Still requires body coordination
-
Multi-task for feet
-
Relatively big to carry around
-
Not suitable for stage performance
-
Need to get use to user interface
iRig Blue Turn
-
Hands-free
-
Easy to carry
-
Wireless connection
-
Affordable ($80 -100)
-
Requires body coordination
-
Multi-task for feet
-
Not compatible with all apps
Machine Learning
-
Hands-free; Eye-tracking
-
Automation
-
High accuracy
-
Not for individual as a state of art
-
Not applicable in daily usage
-
High cost
-
Requires user eye gaze
Insights
Ideal Product Requirements
Body Liberation
Body liberation includes both hands-free and feet-free to minimize body coordination requirement.
Seamless Transition
The incorporation of this technology should minimize effort and smooth the transition.
Conveniency
The product should be easy to carry and to operate.
Ideation
Affinity Mapping
We spent a few days after research to brainstorm different ideas and invited 12 classmates to share their thoughts on designing this product. Later we gathered all the post-it notes and created affinity mapping to categorize input information.
Idea Exploration
From our early brainstorm and affinity map, we further explored each direction of our idea by delving into the world of electronics. Via research and comparison across different possible sensors, we had a better idea about which one would be the most fitting for our poject.
Converge: Let's go with EEG sensor!
Eletroencephalography (EEG) records an electrogram of the electrical activity on the scalp which represents the macroscopic activity of the surface layer of the brain underneath. Some advantages include:
Immediate Reaction
A page flip with a bad timing can easily cause perceivable interruption in a music piece.
Easy to Incorporate
A delayed note causes the rhythm of the whole performance to be compromised.
Non-intrusiveness
Preparing to flip pages at the end measure requires extra mental energy from the musician.
Incorporating Headband
In order to sense signals in the most efficient way as informed by our storyboard, we decided to use Muse headband - a wearable brain sensing headband that measures brain activity via 4 EEG sensors. As a commercialized product, it is relatively stable, and the raw data is easy to be accessed and incorporated into our project.
Storyboarding
We used storyboarding technique to shape the user journey for our target audience. Through sketching, it helps us to understand how the persona goes through each stage of action and clarifies what are the most crucial features for this product.
Prototyping and Testing
Circuit Construction
The circuit consists of the following components:
-
Muse Band 2014;
-
Relay module;
-
6.35mm jack;
-
Arduino Uno Microcontroller;
-
Digital Keyboard;
-
Mobile phone;
-
Laptop.
First Iteration
For the first attempt, we tried to work out a basic circuit as the "Minimum Viable Product" by using the action of blinking to control the sustain pedal. A relay module and 6.35mm jack was incorporated into the circuit.
Insights From The First Iteration
1.
Tilting Instead of Turning Head
We initially used turning head left/right to match up with page flip action. After first-round user testing and feedback, we decided to detect tilting head instead of turning, since it is more ideal for players to keep their eye on the sheets whiling turning head to the other side might cause interruption.
2.
Control Sustain Pedal
We want to liberate our feet from sustain pedals too! For beginning players it is sometimes difficult to coordinate hands with feet. Introducing the actions of blinking or noding elicits a clicking sound generated by the relay and it serves as a metronome!
3.
Detection of Noding
Our initial code used absolute value to detect the action of noding, but limiting to a certain height without calibration is not ideal nor practical. For the second iteration, we were comparing the height of a previous state with a current state to calculate the difference of height in detecting the action of node. Thus, the hand band is calibrated according to user's different head position.
First Iteration: Blinking to press on the sustain pedal
Challenges
Final Prototype
Our solution achieves the two following functionalities:
1. Flip Pages
User can tilt one's hand towards either left or right to control page flipping (to the previous page or the next page).
2. Control Sustain Pedal
User can choose to either node, or blink or control the sustain pedal. These controls are more intuitive via Brain Computer Interface.
Introducing MUSEC
Prototype Demo
Evaluation
Design and Evaluation Metrics
-
Wearability: The wearability of a device is evaluated through placement, human movement, body movement, comfort level, adaptability and perceived influence. Currently, MUSEC satisfies these subdivided metrics by providing direct and intuitive coordination and engagement.
-
Usability: The usability of a device is focused on the functionality, durability, safety and effectiveness. MUSEC is able to conduct all the actions programmed into the wearable device and assist performs to fulfill target actions.
-
Aesthetics: The overall aesthetics of MUSEC maintains a minimalistic style, aligning with the simplicity requirement to maximize convenience for the users.
Limitation
Time and Budget Concerns
Due to the time constraint of this project (3 weeks), we were unable to conduct user testing with MUSEC after the second iteration. However, having a second round of user testing from target users would be ideal for the evaluation and further the future development.
Key Learnings
Maximize Available Resources
We borrowed a MUSE handband from the Digital Futures faculty - great resources! With limited time and budget, we realized the importance of maximizing what is available around us and utilizing it well. Through effective resource allocation and collaboration, we can move project forward much faster, especially for making rapid prototype and quick iterations.
Don't Be Biased By One's Own Assumption
As a designer, personal perspective and motivation are essential for the start of an inspiring project. However, it is also easy to be biased by one's own experience sometimes. We should broaden up the research scope to intake multiple perspectives from identified target users and find out what is the mostly identified common theme, which might not be the same as designer's own interpretation.
Future Development
Alternative Modes of Input
To further this project, we would love to go beyond existing hardwares and ideally explore our own system of input. Undoubtedly, commercialized product has a greater extent of stability which is reliable for prototype delivery and presentation. Nonetheless, designing our own input circuit would allow more flexibility in terms of bio-signal selections and opens up more possibility. Currently, Muse band supports detecting actions such as noding, blinking, jaw-clenching, while with our own sensors, we are able to train the models into detecting other kinds of actions not just limited to the head area.