Real-Time Hand Gesture Recognition for American Sign Language Using CNN, Mediapipe and Convexity Approach

Abstract

Hand gesture recognition is crucial to computer vision and human-computer interaction(HCI), enabling natural and intuitive inter- actions between humans and machines. Sign language is essential for communication between deaf-mute and other people. In this research paper, we proposed an advanced deep learning-based methodology for real-time sign language detection designed to address communication disparities between individuals with normal hearing and the deaf com- munity. The inception of our study involved the creation of a dataset comprising 37 unique sign language gestures. This dataset underwent preprocessing procedures to optimize its relevance for subsequent training of a customized Convolutional Neural Network (CNN) model. The trained CNN model demonstrated proficiency in recognizing and interpreting real-time sign language expressions, thereby contributing to the mitigation of communication barriers in this context. The implementation of two alternative methodologies was executed in parallel with the proposed approach. Specifically, real-time detection of American Sign Language (ASL) employed MediaPipe and Teachable Machine, while the second method illustrates the real-time detection of hand gestures utilizing a convexity-based approach. In this study, the proposed customized CNN model achieved an accuracy of 99.46% and, precision of 99 %, whereas the model trained with the teachable machine achieved 94.54% accuracy, and the method using the convexity approach achieved 84.44% accuracy.