The challenge

The challenge assigned by the course Advance Programming of Interactive System was to use cutting-edge technologies to create innovative user interactions within an application in a 4 week timeframe.

The Solution

• Web App: Created a web app for Chromium browsers with webcam and microphone support for music creation using gestures and voice commands.
• Smartwatch App: Developed a smartwatch app for heartbeat communication and tactile music tempo feedback.
• Gesture Recognition: Used MediaPipe by Google for accurate hand gesture interpretation.
• Voice Commands: Integrated voice commands for touch-free music control.
• Real-time Feedback: Provided real-time feedback for gestures and voice commands.
• Cross-Platform: Ensured compatibility with desktops, laptops, and smartwatches.

Brainstorming at Université Paris-Saclay on the design of the app

My Role & Contributions

• UI Designer & Developer: Drawing sketches of the applications and defining the features of the application.
• Architecture Definition: Defining the architecture of the application after conducting a technology research benchmark of different tech stacks and libraries.
• Feature Development: Coding key features of the application such as user interaction through gesture recognition and the music flow control and the UI of the application.
• Performance Reporting: Creating reports of the app performance with Google Analytics that will drive changes based on user behaviors.

Design Flow

1. Defining the Topic: Thanks to brainstorming sessions, we decided to focus on creating music beats and controlling the music flow.
2. Cutting Edge Technologies: To define the architecture of the app, we based our decision mainly on the performance, the adaptability with other libraries, the community, and the documentation of the technology. We took the path of a computer vision and speech recognition app that can be used from desktops and smartphones and that also connects with Android smartwatches.
3. Unexplored and Innovative Application: Combined hand gesture recognition with music control and beats creation, and receiving live tactile feedback on the watch.

Application Design

The initial design phase focused on creating a UI and interaction system for a music application using hand gestures and speech recognition. We explored two primary design approaches:

1. DJ Console Interface: We considered a familiar layout resembling a DJ console, which would leverage users' existing mental models of music creation tools.
2. Innovative Interaction Design: Our goal was to enable users to create beats through simple, intuitive gestures like a two-finger touch. After three low-fidelity prototype iterations, we finalized the UI design.

Gesture Mapping and Implementation

We utilised the Mediapipe API to analyse hand tracking capabilities and selected specific gestures for action mapping. The primary challenge was designing a system that could initiate actions from any user position. Our solution was implementing a finite state machine that dynamically interprets gesture sequences to trigger specific actions.

• Standard Mode: 4 drumpad sounds
• Christmas Mode: 4 seasonal-specific sounds
• Piano Mode: 4 piano chords for foundational musical composition

Learnings

• Experience with OpenCV, Mediapipe and Webkit libraries for hand gesture and speech recognition.
• Designed the architecture of the communication of the Web app with the Smartwatch using network sockets under the same internet connection and exploiting smartwatch features.
• Valuable learning about the usability and discoverability of the application's features were collected thanks to user feedback.

Results

• Web application designed, implemented, and launched in a 4-week timeframe.
• The “wow effect” of the application created a word-of-mouth effect that brought 60 new users per month to try the platform.
• Development of personal skills: User Interactions design, Web and Smartwatch development, Application tracking, and User testing.
• Live demo at Université Paris-Saclay.
• Live demo and interview at Good Soup Media channel.