let's make music more fun and accessible 🙌
ABOUT ME
Music has been a big part of my life. I've spent years studying and playing piano, music theory, and history, and have also played flute, sung in choirs and a cappella groups, taught piano and music theory, and more recently started learning guitar.
All those years of playing and studying music shaped the way I think about harmony. I tend to organize musical ideas visually and intuitively in my head, and that way of thinking eventually became the foundation for Gesture Synth.
MY MAIN PROJECT
Gesture Synth is a browser-based musical instrument created by Eric Wei that uses hand tracking to let you play chords with your hands while singing the melody yourself.
I designed it around the idea that the instrument should handle the harmony while you provide the melody with your voice. In that sense, it's a little more like a ukulele than a piano: you use simple gestures to move through useful chords while you sing.
I had seen hand-tracking instruments and vocoders online, but many of the musical interfaces I came across treated gestures like buttons in the air: make one gesture and trigger one sound.
I wondered whether hand tracking could be used for something more expressive, especially for chords. Rather than pressing invisible buttons, I wanted the gestures themselves to represent musical relationships.
I quickly realized this new medium unlocked many expressive possibilities. Hands can create different shapes, move in three dimensions, and tilt in different directions. I wanted to use those capabilities to create a system where the gestures themselves represented musical relationships.
The result was a system where your hands navigate harmony instead of simply triggering individual notes.
THE JOURNEY
From a local experiment to an instrument people around the world could try.
JULY 6, 2026
I posted my first video of Gesture Synth while the app was still running locally on my computer. The core gesture system was already there. Initially, the right hand controlled chord density; I later changed it to control chord voicings for more musical expression.
Watch the first video ↗JULY 11, 2026
I deployed Gesture Synth through Vercel, so anyone could try it in their browser without setting anything up locally.
Try it in the browser ↗JULY 19–22, 2026
Performances of Like a Prayer, Hit the Wall, and Avicii’s The Nights helped bring Gesture Synth to a much wider audience.
JULY 19
A performance that helped bring Gesture Synth to a much wider audience.
Watch video ↗JULY 23, 2026
After people asked how to try it themselves, I published a tutorial walking through how to play Gesture Synth.
Watch the tutorial ↗JULY 26, 2026
I made the source code available on GitHub so other people could explore the project and learn from how it works.
Explore the source code ↗HOW IT WORKS
Gesture Synth uses hand tracking to turn simple hand movements into chords and musical expression. The left hand chooses the chord, while the right hand controls how it sounds.
LEFT HAND
Fingers → chord degree
Tilt → major or minor
Tilt your hand inward for major and outward for minor. Move between the two harmonic worlds without changing the chord shape.
Watch the left-hand-only tutorial ↗
RIGHT HAND
Fingers → chord voicing
The thumb changes the chord's register.
Position → expression
Move your hand higher or lower to control volume. Tilt your hand to sweep the filter and change the character of the sound.
Watch the Gesture Synth tutorial ↗
One of the best parts of sharing Gesture Synth has been seeing other people experiment with the idea and make it their own.
Creators including Liv Angell, saaisha.music, and Rafferty Coope have explored gesture-based music-making in their own ways.
The source code is also available on GitHub for people who want to learn from the project or build on it.
FAQ
A few things to know about how Gesture Synth works and how to get started.