AI-Based Arabic Sign Language to Voice Translation System

Abstract

The Deaf and Hard-of-Hearing (DHH) community in the Arab region faces major communication difficulties arising solely because of the scarcity of interpreters. Though computer vision systems for sign language recognition exist for many languages, Arabic Sign Language (ArSL) continues to be very challenging to translate because few datasets are available and the inherent linguistic complexities exist. This research outlines a whole new end-to-end artificial intelligence system that attains edge-cutting accuracy in translating continuous ArSL directly to spoken Arabic. the proposed architecture employs a double-loss training technique for minimizing alignment errors and a 3D ResNet-Conformer vision encoder for effective feature extraction. Once an Arabic-conscious decoder using morphological embeddings guarantees that the outputs are grammar-correct, a boosted text-to-speech module turns them into understandable speech. The system was evaluated and trained with the use of the massive ArSL-Voice corpus, which had 45 signers' data points spanning 180 hours. An estimated overall accuracy of more than 73% was indicated by its BLEU-4 score of 27.5 and Word Error Rate (WER) of 28.7%, demonstrating remarkable efficacy. Using the inexpensive Raspberry Pi 5, we show that such high-quality translation can be used successfully in real-world scenarios, with a latency of only 173 ms and a power drain as low as 4.1W. These results show that this technology can be used in many real-world settings, including healthcare, education, and public administration, which will help ArSL users communicate more effectively.