🧠
Project RAIN
Project RAIN is an interactive music rhythm game with hand movements. By wearing the ring-like wearable interactive device X Ring, the player is supposed to make various designated gestures according to the music rhythm and screen prompts. The game will give scores based on the accuracy of the player’s gestures. The player’s goal is to achieve a higher score and fully enjoy the music in the game.
Project Type
Game with alternative controller
🔧
🛠
Time
Jan 2022 - Jul 2022
My Role
Collected and labeled motion data for seven distinct two-handed gestures. Trained a custom recognition model to classify gestures in real-time with high accuracy.
Designed and assembled the wearable input device using inertial sensors to capture gesture motion. Integrated a Bluetooth Low Energy (BLE) module to wirelessly transmit data between the ring device and computer.
Industrial Design & Fabrication
Modeled the ring hardware using OpenSCAD and produced the physical prototypes via 3D printing for functional testing and gameplay use.
Project Brief
Project RAIN is a gesture-controlled rhythm game developed in Unity. The name is a playful homonym of “ring,” referencing the custom-built X Ring — a wearable input device that detects real-time hand motions.
The game is designed to be played with intuitive two-hand gestures, offering a flexible, immersive music experience without requiring a large physical play area. Players navigate menus and perform in-game actions using natural motions — for example, turning the right hand upward or downward moves the menu cursor, while slapping both hands together confirms a selection.
Once a track is selected, the player must perform specific gestures to interact with colorful cubes flying toward the screen in sync with the music. Each cube’s shape and color indicates a different gesture, and timing is guided by a pulsing visual ring. The closer the gesture is to the correct moment and movement, the higher the player’s score.
Notes
Visual assets were partially sourced from the Unity Asset Store, while gameplay logic, gesture integration, and input handling were fully custom developed.
Gameplay video
X Ring is a custom-built, wireless wearable device worn on both hands, capable of recognizing seven distinct hand gestures in real time. The ring form factor was chosen for its lightweight design, comfort, and visually appealing appearance — offering players an intuitive and unobtrusive way to interact with the game.
Players simply plug a mini-USB receiver into their computer — similar to a wireless mouse — to establish the connection. No additional configuration is required.
Under the hood, X Ring uses a miniature development board compatible with the Arduino IDE. It includes a six-degree-of-freedom inertial measurement unit (IMU) to capture gesture motion data, and a Bluetooth Low Energy (BLE) module to wirelessly communicate with the game system.
To classify gestures, a Convolutional Neural Network (CNN) model is trained using gesture motion data collected during development. The trained model is converted into an Arduino-compatible C++ library using TensorFlow Lite, allowing it to run directly on the ring's embedded hardware. When a gesture is performed, the onboard model classifies the motion and sends the predicted label — rather than raw data — to the receiving computer via BLE. This significantly reduces communication latency and offloads processing from the game runtime.
The ring’s hardware casing is modeled in OpenSCAD and 3D printed using black resin for durability. The finger-contact area is covered in soft rubber to improve comfort during extended use.
Gesture Set
The game supports seven unique gestures, each mapped to in-game note types:
Turning the hand upward
1.
4.
Pushing the hand forward
2.
7.
About X Ring
Turning the hand downward
5.
Slapping both hands together
Slapping the shoulder
My Role
Individual project
3.
6.
Drawing a clockwise circle in the air
Slapping the back of both hands
Design Inspiration
Project RAIN was inspired by Just Dance by Ubisoft — a game I enjoyed, but often found limited by the physical space it required. I wanted to create a more flexible, gesture-based music game that could be played comfortably, even at a computer desk.
When listening to music, people naturally respond with physical movements — clapping, waving, or gesturing along with the beat. These spontaneous motions deepen our connection to rhythm and help us feel the music more fully. Traditional rhythm games typically rely on keyboard or touchscreen inputs, which often fall short of delivering an immersive experience.
With Project RAIN, I aimed to enhance musical immersion by translating intuitive hand gestures into gameplay. Using a custom-built gesture ring, players can interact with rhythm in a more expressive, physical, and engaging way — turning movement into music.
Gesture Recognition Algorithm
Hardware Prototyping
Overview
🎮
Gameplay Programming
Designed and implemented all game logic, including UI behavior, score calculation, and note generation based on configurable JSON patterns.
Platform
Unity, Arduino