Interaction Design | UX Research | Mixed Reality
Design of a Multimodal Mixed Reality Work Environment with Wearable Technology
Team
Individual thesis project under the supervision of Dr. ALexis Morris and Dr. Claire Brunet
My Role
Research, Literature Review, Concept development, Prototype Design and Development, User Testing, Evaluation
Timeline
August 2022 - April 2023
Tools
Arduino Programming Language (C++), Arduino Uno, PPG Heart Sensor, Temperature Sensor, GSR, Ultrasonic Sensor HC-SR04, Unity (C#), Oculus Quest 2 and Quest Pro
Project Overview
Issues relating to health and well-being at work have risen in prominence, exerting negative effects upon both individuals and organizations. Two main contributing factors are a lack of awareness of one's bodily status and a lack of accessible and effective adjustment mechanisms.
This study proposes and develops a hybrid wearable and Mixed Reality system prototype that enhances awareness of bodily status and provides mediation. This prototype can adapt to the individual's real-time biometric data through a wearable glove, and provide personalized feedback in a Multimodal Mixed Reality working environment.
Problem Space
Workplaces in the majority of contemporary society are desk-based, with extensive sitting and little exercise during working hours. Given that physical activity can have positive impacts on musculoskeletal pain, depressive symptoms and anxiety, and work performance outcomes, this lack of physical activity and movement among desk-based workers is concerning (Ryde et al., 2020).
Identified Challenges
Key factors that contribute to the problem include: 1) Inadequate or non-existent health and safety policies; 2) Poor communication and management practices; 3) Lack of support for employees; 4) Work performance pressure; 5) Job insecurity; 6) Lack of psychological safety; 7) Negative workplace environment.
Key Problem 1
Lack of awareness towards one's own bodily status while working;
Key Problem 2
Lack of accessible and affordable adjustment mechanisms.
Research Question
How can the design of a Multimodal Mixed Reality Work Environment enhance body awareness and provide mediation?
Research & Design Process
The iterative design process starts with 1) Analysis: I identified user demands and figured out workable solutions through the exploratory research guided by the Interaction Design principles; 2) Projection: I designed and developed workable prototype which attempts to provide a solution towards identified problem; and 3) Synthesis: I reflected on the usability and functionality of the designed prototype and envisioned new iterations, ranging from low-fidelity to high-fidelity prototypes.
Literature Review
The literature and contextual review for this research builds on the interdisciplinary fields of physiology and biosensor research, Mixed Reality, and Environmental Psychology. Each of these fields has substantive studies related to health monitoring and visualization. By exploring each of these areas, this research aims to identify their potential to inform the biofeedback-informed MMRWE.
Environmental Psychology
Physiology and Biosensor Research
Mixed Reality Technology
Exploratory Research: Need-Finding
User Personas
Empathy Mapping
Synthesis
System Diagram
The proposed MMRWE is designed to enhance users' awareness of bodily changes and support meditation. The system includes two main components:
1). Biometric signal collection and processing via a wearable glove and
2). Multimodal MR output via a Head Mounted Display (HMD).
Low Fidelity Prototypes
Paper and Wizard of Oz Prototypes
These early-stage prototypes employ a minimalistic design, which aims to reduce distraction and minimize interference with the user at a working status. The design approach and aesthetics adhere to the calm technology principles, which aims to create technology that is minimally invasive, informative, and calming (Case, 2015).
The use of a grid structure to align visual elements is a deliberate reference to the traditional desktop interface, which provides a familiar and intuitive experience for users. This approach also aims to effectively organize complex sets of information, as it allows users to quickly scan and locate specific functions.
User Interface Prototypes
This interface design features four icons on the left side of the screen that represent four biometrics: heart rate, respiration, body temperature, and GSR. On the right-hand side of the screen, an animated text prompt appears alongside audio output. The triggered output based on biometric signals are designed in an minimalistic and non-invasive way, such as a breathing guide that is triggered only when the user's heart rate raises abruptly. Compared to the previous paper and Wizard of Oz prototypes, this iteration reduces the visual complexity of synthesizing multiple bodily information at once, and outputting single feedback at once to the user.
Biometric Sensor Mapping and Wearable Prototype
Signal Mapping
This stage involves selecting the most appropriate sensors for detecting bodily changes, such as the GSR sensor, PPG heart sensor, temperature sensor, and ultrasonic proximity sensor. These sensors are then integrated into a wearable glove, which is designed to provide users with real-time feedback on their bodily changes while allowing for maximum hand and finger movement.
Wearable Glove Sketches
Circuit Diagram and Glove Construction
The sensors are connected to an Arduino Uno board, which can be powered either by a laptop and a cable or by a 9V battery that can be attached to the glove and communicates via WiFi. The proximity sensor is attached to the back of the chair, and the collected data can be transmitted to the MR environment for output.
The wearable glove was constructed by placing the GSR electrodes on the index and middle fingers, heart rate sensor on the little finger, and temperature sensor on the ring finger. A half finger glove was chosen in order to refrain from limiting the user's finger dexterity, which can be crucial for operating at work or typing on a keyboard.
MR & VR Design Sketches
Design Iterations
The four MR design sketches explore the potential of wearable and MR technologies to create immersive and responsive environments that enhance individual awareness and provide mediation. Each sketch explores different aspects of this potential, from physiological sensing and feedback in Design Sketch 1, to the use of natural elements to symbolize bodily status in Design Sketch 3, to the multimodal approach and virtual environment of Design Sketch 4. While each iteration represents a varied design approach, all share a common goal of creating an immersive, responsive, and intuitive user experience.
Design Sketch 1 - 4: Workstation Enhancement (ShapeXR); Embodied Natural Elements; Virtual Workstation Environment (Unity)
Final Prototype: MMRWE
This final prototype represents the culmination of the research, design, and testing conducted in this thesis study. The MMRWE design aims to provide real-time reflection of biometric signals and integration between the physical and digital worlds to enhance awareness of bodily changes and provide mediation. The user interface of the wearable and MR display allows for both tangible and virtual interaction with the prototype, providing an additional layer of dimension to the user's workspace.
MR Documentation
Evaluation
Evaluation Criteria
The evaluation of the final prototype employs RtD and ID methodologies. RtD involves creating and refining design artifacts through a continuous process of analysis, projection, and synthesis, where each iteration of design is informed by user feedback and reflection. In this study, the evaluation of the wearable glove and MR environment served as an integral part of the iterative design process, aimed at measuring the effectiveness of the system in enhancing bodily awareness and providing mediation.
Seven participants were enlisted into the user study, of which 4 are female and 3 are male. The target participants were workers who are familiar with office settings, including graduate students, part-time and full-time office workers. The average age was 28 years old (M = 30.4, SD = 11.5). The user study was REB approved and written consent was obtained from all the participants.
EC for Wearable
1. Comfort
2. Reliability
3. Functionality
4. User Satisfaction
EC for MR system
1. Comfort
2. Effectiveness
3. Immersion
4. User Satisfaction
User Testing Procedure
The user study was conducted in a lab workspace during the typically hours of the workday. Each user study was carried out individually with an assigned time lot. After a brief introduction and acknowledgement of this study, participants proceeded the session consisted of 1). a 5-minute preliminary survey; 2). a 15-minute prototype interaction with unguided or guided instructions; 3). a 5-minute post-study survey; 4). a 30- minute semi-structured interview. Participants were instructed to put on wearable glove and HMD (Oculus Quest Pro) during the experiment.
Feedback and Results
The results of the user study surveys conducted for this study indicate that the prototype enhanced participants’ awareness of their health and bodily status significantly. The proportion of participants reporting a high level of awareness increased from 25% before the testing to 57.14% after the testing. Participants’ experiences with the wearable glove and MR environment were generally positive, with 85.71% and 71.43% rating the experience as positive or very positive, respectively. Moreover, participants reported feeling calmer and more emotionally satisfied after experiencing the prototype.
The functionality of the wearable receives a score of 0.679 (Min-max normalization), while the comfort of the MR system receives a score of 0.571. These lower scores can be attributed to the lack of robustness in the selected sensors and the side effects of dizziness and lagging associated with MR devices. However, despite these concerns, the overall user satisfaction remains high, particularly in terms of the comfort level and dexterity provided by the wearable glove.
Through semi-structure interview, several prominent themes emerged, shedding light on various aspects of the user experience. The analysis of the interview data revealed several high frequency words, such as ”be placed further away,” ”calming,” ”smaller fans,” ”dizziness,” ”focus and reduce disruption,” ”comfortable,” ”subtleness,” and ”dynamic lighting.” These words indicate a preference for a more calming and less intrusive visual and multi-modal design, while shedding light on the side effects and disadvantages currently associated with MR devices.
Exhibition
An on-site exhibition took place in April 2023 as part of the graduate thesis exhibition. The prototype was showcased and experienced by visitors. This three-day exhibition has helped me gain more perspective about public's response towards the design and offered valuable insights in terms of future development.
Conclusion
This project aimed to design a MMRWE with wearable technology that enhances bodily awareness and promotes relaxation in work environments. The project identified a research gap and aimed to contribute to biofeedback-informed MR applications for work scenarios.
Overall, this working prototype serves as a proof-of-concept that highlights the potential of wearable and MR technology to enhance individual awareness and mediate potential negative consequences induced by work. With further development and refinement, MMRWE has the potential to become a valuable tool in various industries to promote a healthier and more productive work environment.
Revisiting Contributions and Goals
The key contributions of this project include the introduction of a prototype system that dynamically mediates its environment based on real-time monitoring of users' physiological signals, the incorporation of principles from Environmental Psychology in the design of MR technology and human-computer interaction, and an investigation of the possibility of future workspaces design in a multimodal MR setting.
The small-scale user testing conducted provided positive feedback and generated valuable insights into how biofeedback-informed MR applications can be designed for work scenarios. Further research could build upon this study by engaging a greater number of audiences and applying the design mechanism to other scenarios.
Limitations
This study has several limitations and constraints that must be acknowledged. These include limitations in hardware resources, data acquisition and processing and user-testing.
Hardware limitations relate to HMD (Oculus Quest 2/Pro), which was used for this study. While the headset provides an immersive and engaging experience, it also has potential effects on the human body, such as motion sickness and eye strain. Additionally, the headset's comfort level and accessibility may be limited for some users. This could affect the overall effectiveness of the system and may reduce its potential impact.
While the PPG heart sensor, GSR sensor, TMP36, and Ultrasonic Proximity sensors are all effective tools for physiological monitoring, they do have some limitations. Data acquisition and processing needs to be evaluated in terms of sensor stability and accuracy. There may be some discrepancies between the measured data and the actual physiological responses of the user.
Future Development
The biofeedback-informed MR system project is based on the assumption that MR technology and headset will become more affordable, comfortable, and accessible. To remain relevant and effective as technology advances, future development should consider the development of relevant technology and be prepared to alter design decisions based on alternative hardware and form factors. Wearable sensors and other hardware used in this study can be replaced by more stable and developed products that would boost the accuracy of signal input and effectiveness.
The potential applications of the developed system are vast and varied, extending beyond workspaces to healthcare facilities, rehabilitation centers, assisted living communities, and in-home use for patients and aging populations. By expanding the scope of this project to include different populations and environments, it has the potential to transform the way we interact with our surroundings and enhance our overall well-being.
Personal Reflection & Final Remarks
In conclusion, this project presents a promising contribution to the field of wearable technology and Mixed Reality systems for body awareness enhancement and mediation. The project's innovative design framework and insights from user testing and feedback provide a foundation for future research and development in this area, going beyond defined problem space.
Throughout this thesis study, I gained a wealth of knowledge and skills that have proved invaluable in both my personal and professional development. I deepened my understanding about biometric sensors and their applications, particularly in wearable design and human interaction. I also expanded my programming skills, particularly in Unity using C#. As someone from a non-technical background, this aspect of the project was initially challenging, but through self-learning, utilizing available online resources, and the help of my friends and supervisor, I was able to make significant progress.