~/project/hakotenfruits.md
Hakoten Fruits
Fruit Ninja with your hand, no headset needed
- Timeline
- December 2021 - January 2022
- Type
- Side-Project
## overview
VR games like Beat Saber are a great way to get exercise, but VR headsets can be expensive. Hakoten Fruits is a game inspired by Fruit Ninja that uses your webcam to track your hand, so your hand becomes the sword you use to slice fruit.
## how-it-works
How it works
MediaPipe finds 21 points on your hand in each frame from the webcam. I take two of them, your wrist and the base of your middle finger, and use the point halfway between them as the centre of your palm. That point moves the mouse, and the sword follows the mouse. So the game itself is just a normal Pygame game, and it doesn't need to know a hand is controlling it.
I picked the palm on purpose. Fingertips move a lot and disappear when you close your hand, so a sword tied to a fingertip shakes around. The palm stays steady while you swing.
A couple of small details made it feel right. The camera image gets flipped first, otherwise moving your hand right would move the sword left. The game window is also the same size as your camera, so the edges of the screen match what the camera can see.
## when-it-loses-your-hand
When it loses your hand
Your hand leaves the camera a lot when you're swinging at fruit. Instead of letting fruit fall while the sword is stuck, the game shows a message that it can't find your hand and pauses for half a second so you can bring it back.
There's also a small window that shows what the camera sees, but only as the outline of your hand on a black background, shrunk to a quarter of the size. You can tell right away if it's tracking you, without it covering the game.
## the-game
The game
Fruit falls in waves, and each time you clear a wave, the next one has more fruit. Each fruit is rotated randomly so they don't all look the same, and once in a while a coconut shows up and falls faster than the rest. I drew all the art in Pixilart.
## key-learnings
What I learned
- 01
Picking the right point to track mattered more than the model. The palm made the same model feel a lot steadier.
- 02
Plan for the model losing track. A short pause and a clear message felt better than a stuck sword.
## built-with
Languages & libraries
- Python
- Pygame
- OpenCV
- MediaPipe
- NumPy
Tools
- VS Code
- Pixilart
## side-quests
That's all the projects
Up next: trips, team photos, and moments with friends.
see the side quests