Project Gameface: A hands-free, AI-powered gaming mouse

Project Gameface is a new open-source hands-free gaming mouse that enables people to control a computer's cursor using their head movement and facial gestures. People can raise their eyebrows to click and drag, or open their mouth to move the cursor, making it possible for anyone to pursue gaming.

The project was inspired by the story of quadriplegic video game streamer Lance Carr, who lives with muscular dystrophy, a progressive disease that weakens muscles. Lance refused to let his physical limitations keep him from his love of gaming, so he used a head-tracking mouse to control his cursor. In 2021, Lance's house caught on fire while he was gaming one night — and it was all caught on a live stream and watched by gamers around the globe. Lance lost all of his online gaming equipment in the fire, including the expensive head-tracking mouse, which is the primary technology that enabled him to play computer games.

After hearing Lance's story, we knew we had to help. So we worked with Lance to co-design Project Gameface, a hands-free gaming mouse that uses Google MediaPipe open-source solutions. Head movements and facial gestures captured simply from a webcam are recognized by machine learning models to move the cursor accordingly. To adapt to the needs of different users, we incorporated the idea of a gesture size, which allows users to control how prominent their gesture needs to be to trigger a mouse action.

Project Gameface is a promising new technology that has the potential to make gaming more accessible. The code is open source and available in preview today. The tool is still in development, but we're excited about the potential it has to change the lives of people because it’s relatively inexpensive to build and maintain. If you're interested in contributing, improving, or using this technology, visit our GitHub repository.