Abstract
This paper subverts the traditional understanding of Photoplethysmography (PPG) and opens up a new direction of the utility of PPG in commodity wearable devices, especially in the domain of human computer interaction of fine-grained gesture recognition. We demonstrate that it is possible to leverage the widely deployed PPG sensors in wrist-worn wearable devices to enable finger-level gesture recognition, which could facilitate many emerging human-computer interactions (e.g., sign-language interpretation and virtual reality). While prior solutions in gesture recognition require dedicated devices (e.g., video cameras or IR sensors) or leverage various signals in the environments (e.g., sound, RF or ambient light), this paper introduces the first PPG-based gesture recognition system that can differentiate fine-grained hand gestures at finger level using commodity wearables. Our innovative system harnesses the unique blood flow changes in a user's wrist area to distinguish the user's finger and hand movements. The insight is that hand gestures involve a series of muscle and tendon movements that compress the arterial geometry with different degrees, resulting in significant motion artifacts to the blood flow with different intensity and time duration. By leveraging the unique characteristics of the motion artifacts to PPG, our system can accurately extract the gesture-related signals from the significant background noise (i.e., pulses), and identify different minute finger-level gestures. Extensive experiments are conducted with over 3600 gestures collected from 10 adults. Our prototype study using two commodity PPG sensors can differentiate nine finger-level gestures from American Sign Language with an average recognition accuracy over 88%, suggesting that our PPG-based finger-level gesture recognition system is promising to be one of the most critical components in sign language translation using wearables. I. INTRODUCTION The popularity of wrist-worn wearable devices has a sharp increase since 2015, an estimation of 101 4 million wrist-worn wearable devices will be shipped worldwide in 2019 [1]. Such increasing popularity of wrist-worn wearables creates a unique opportunity of using various sensing modalities in wearables for pervasive hand or finger gesture recognition. Hand and finger gestures usually have a diverse combinations and thus present rich information that can facilitate many complicated human computer interaction (HCI) applications, for example wearable controls, virtual reality (VR)/augmented reality (AR), and automatic sign language translation. Taking the automatic sign language translation as an example illustrated in Figure 1, a wrist-worn wearable device (e.g., a smartwatch or a wristband) could leverage its sensors to realize and convert sign language into audio and text and back again, which will greatly help people who are deaf or have difficulty hearing to communicate with those who do not know the sign language. Existing solutions of gesture recognition mainly rely on cameras [2] [4] microphones [5], [6], radio frequency (RF) [7] [9] or special body sensors (e.g., Electromyography (SEMG) [10], Electrical Impedance Tomography (EIT) sensor [11], and electrocardiogram (ECG) sensor [12]). The approaches using cameras face occlusion and privacy issues.