top of page
In this project, we investigated the potential of robot arms to enhance sports and gaming training in a simulated environment. In the physical world, expertise in sports and games usually entails repetitive exercise and access to appropriate equipment. Unfortunately, such resources may not be accessible to everyone, which is where robot arms come in as a valuable aid. By generating virtual scenes and offering support for sports or game-based training, robot arms can be of tremendous assistance to a wider population.
Interaction Design | Game Design
Exploring the potential of Universal Robots Arms (UR10e) in Game Design
Design team of four
Ideation, Game design, Unity design and development (shared with another team member), Integration
March - April 2023, 4 weeks
Arduino, Arduino IDE, Unity, Light sensor, WebSocket, Other electronic components for circuit construction
The Arduino converts the data into JSON format and sends it to the WebSocket server, which in turn receives the data and sends it to Unity. Unity then converts the received data and convert it into C# format.
Arduino Design and Sensor Container Fabrication
The game is designed to improve players' reaction capacity, an essential skill for developing prompt and responsive thinking. With the assistance of a robot teammate, players of all ages can engage in exciting challenges that promote hand-eye coordination and quick thinking.
Thus, we used a light sensor, by which we can make a physical controller (pointer) that facilitates hands-on and tangible interface for reaction training.
We used laser cutting technology for the container fabrication, some steps in case you are interested:
Simplify its features into lines
Create a vector file of the design using software according to cutting guideline
Choose a timber material for the laser cutting
Game Environment Design
Exploration with VR, and live data
We mapped the coordinates of the tip of the robot arm onto the Unity asset - mask as shooting target. Thus by programming the robot arm movement, the target in the Unity scene moves accordingly.
Step 1: Connect both Arduino and Unity to Websocket
We connected Arduino and Unity to the WebSocket for real-time data transfer, use a light sensor and laser pointer to trigger animations, add a score system, reset button, and LED light strip for feedback. This created an engaging game experience that responds promptly to user actions.
Step 2: Refinement
We refined the scene design and created storylines that will add depth to the game. We also started to consider the design of the Arduino container and cable organization to enhance user experience. Lastly, We changed the scoreboard to a reward system that will make the game more engaging and encourage players to continue playing.
Step 3: Final Adjustment
We laser cut the monster as the target and added ambient light to create a more immersive gaming experience, then we programmed the robot arm to respond to user’s movements, and enhancing the design of the Arduino container. These improvements allowed a more seamless and enjoyable game.
In order to prevent the Arduino from sending an excessive amount of data to the WebSocket server, which could result in overwriting data and causing Unity to refresh the score too quickly, we conducted several tests to determine the optimal delay time. Based on our findings, we have determined that a 2-second delay is the ideal time for Unity to refresh the score. However, this delay has created some issues on the Arduino side, as it causes a delay in providing timely feedback to the user about whether they have hit the sensor. This, in turn, has had a negative impact on the overall gaming experience. We will use better design to Prevent users from perceiving delays in the future.
This project explores the potential of robot arms in enhancing sports and gaming training by combining physical objects with virtual environments.
Some future directions we think would be cool to develop towards include:
Integration with Mixed Reality environment: enhance immersive and interactive experiences, and expand the range of training scenarios.
Personalized training program: incorporate sensors and ML algorithms to help robot system adapt to users' needs over time, enabling users to focus on areas where they need improvement and develop skills in a more targeted way.
Credit to my team members:
Wentian Zhu, Ellie Huang, Jiamin Liu, Yifan Xu;
and the support from DF OCAD faculty and resources.
bottom of page