top of page
user flow.jpg
Screen Shot 2023-05-10 at 9.00.46 AM.png
Project Overview
In this project, we investigated the potential of robot arms to enhance sports and gaming training in a simulated environment. In the physical world, expertise in sports and games usually entails repetitive exercise and access to appropriate equipment. Unfortunately, such resources may not be accessible to everyone, which is where robot arms come in as a valuable aid. By generating virtual scenes and offering support for sports or game-based training, robot arms can be of tremendous assistance to a wider population.
Interaction Design | Game Design 
Exploring the potential of Universal Robots Arms (UR10e) in Game Design
Design team of four
My Role
Ideation, Game design, Unity design and development (shared with another team member), Integration
March - April 2023, 4 weeks
Arduino, Arduino IDE, Unity, Light sensor, WebSocket, Other electronic components for circuit construction


design process- connection.jpg
The Arduino converts the data into JSON format and sends it to the WebSocket server, which in turn receives the data and sends it to Unity. Unity then converts the received data and convert it into C# format.
Arduino Design and Sensor Container Fabrication
Screen Shot 2023-05-10 at 9.01.07 AM.png
The game is designed to improve players' reaction capacity, an essential skill for developing prompt and responsive thinking. With the assistance of a robot teammate, players of all ages can engage in exciting challenges that promote hand-eye coordination and quick thinking.

Thus, we used a light sensor, by which we can make a physical controller (pointer) that facilitates hands-on and tangible interface for  reaction training.
Screen Shot 2023-05-10 at 9.01.45 AM.png
We used laser cutting technology for the container fabrication, some steps in case you are interested: 
  1. Simplify its features into lines
  2. Create a vector file of the design using software according to cutting guideline
  3. Choose a timber material for the laser cutting
Game Environment Design
Exploration with VR, and live data
design process-2.jpg
We mapped the coordinates of the tip of the robot arm onto the Unity asset - mask as shooting target. Thus by programming the robot arm movement, the target in the Unity scene moves accordingly.
Design Processes
Step 1: Connect both Arduino and Unity to Websocket

We connected Arduino and Unity to the WebSocket for real-time data transfer, use a light sensor and laser pointer to trigger animations, add a score system, reset button, and LED light strip for feedback. This created an engaging game experience that responds promptly to user actions.
Step 2: Refinement
We refined the scene design and created storylines that will add depth to the game. We also started to consider the design of the Arduino container and cable organization to enhance user experience. Lastly, We changed the scoreboard to a reward system that will make the game more engaging and encourage players to continue playing.
Step 3: Final Adjustment
We laser cut the monster as the target and added ambient light to create a more immersive gaming experience, then we programmed the robot arm to respond to user’s movements, and enhancing the design of the Arduino container. These improvements allowed a more seamless and enjoyable game.
User Flow
Screen Shot 2023-05-10 at 9.01.25 AM.png
Video Documentation
Final Remarks
In order to prevent the Arduino from sending an excessive amount of data to the WebSocket server, which could result in overwriting data and causing Unity to refresh the score too quickly, we conducted several tests to determine the optimal delay time. Based on our findings, we have determined that a 2-second delay is the ideal time for Unity to refresh the score. However, this delay has created some issues on the Arduino side, as it causes a delay in providing timely feedback to the user about whether they have hit the sensor. This, in turn, has had a negative impact on the overall gaming experience. We will use better design to Prevent users from perceiving delays in the future.
Future Directions
This project explores the potential of robot arms in enhancing sports and gaming training by combining physical objects with virtual environments.

Some future directions we think would be cool to develop towards include:

  1. Integration with Mixed Reality environment: enhance immersive and interactive experiences, and expand the range of training scenarios.
  2. Personalized training program: incorporate sensors and ML algorithms to help robot system adapt to users' needs over time, enabling users to focus on areas where they need improvement and develop skills in a more targeted way.
Credit to my team members:
Wentian Zhu, Ellie Huang, Jiamin Liu, Yifan Xu;
and the support from DF OCAD faculty and resources.
bottom of page