STEMM Institute Press
Science, Technology, Engineering, Management and Medicine
Design of a Lightweight Sign Language Recognition Model for On-Device Deployment
DOI: https://doi.org/10.62517/jbdc.202601412
Author(s)
Chenxi Zhang*
Affiliation(s)
Honors College, Ningbo University of Finance and Economics, Ningbo, Zhejiang, China *Corresponding Author
Abstract
This paper studies offline and real-time sign language recognition with a wrist-worn device. The model is selected according to the operators supported by the runtime. Recurrent structures are not used because static unrolling increases the INT8 model size to 988.2 KB; the final design uses five operator types: one-dimensional convolution, depthwise separable convolution, global average pooling, a fully connected layer and Softmax, with 11 366 parameters and an INT8 size of 29.3 KB. On the subject-disjoint split of UCI HAR, accuracy is 92.81% in floating point and 89.16% after INT8 quantization, statistically indistinguishable from a plain one-dimensional convolution with three times the compute (p = 0.579), while an iso-compute variant is 1.72 percentage points higher (p = 0.037). On the ESP32-S3, a single inference takes 15.94 ms with a 37.1 KB Tensor Arena and no PSRAM. On a self-collected 12-class dataset covering two subjects and seven sessions, same-subject, same-hand cross-session accuracy is 82.41% with the sensor left in place and rises to 90.29% after merging two easily confused classes; once the sensor is re-donned, cross-subject, cross-hand and cross-day accuracy falls to 28.94%–38.30%, and recovers to 77.49%–86.90% after gravity alignment and classifier-head fine-tuning using two samples per class. The results show that a model designed for a limited operator set can maintain its accuracy while using about one third of the computing cost. This advantage appears to come more from the model structure than from the computing budget.
Keywords
Sign Language Recognition; Model Quantization; Depthwise Separable Convolution; TinyML; On-Device Inference
References
[1]L. M. Haile, K. Kamenov, P. S. Briant, et al. Hearing loss prevalence and years lived with disability, 1990-2019: findings from the Global Burden of Disease Study 2019. The Lancet, 2021, 397(10278):996-1009. [2]S. Sharma, R. Gupta, A. Kumar. A TinyML solution for an IoT-based communication device for hearing impaired. Expert Systems with Applications, 2024, 246:123147. [3]Y. Gu, C. Zheng, M. Todoh, et al. American sign language translation using wearable inertial and electromyography sensors for tracking hand movements and facial expressions. Frontiers in Neuroscience, 2022, 16:962141. [4]R. David, J. Duke, A. Jain, et al. TensorFlow Lite Micro: Embedded machine learning on TinyML systems. Proceedings of Machine Learning and Systems, 2021, 3:800-811. [5]D. Anguita, A. Ghio, L. Oneto, et al. A public domain dataset for human activity recognition using smartphones. Proceedings of the 21st European Symposium on Artificial Neural Networks, Bruges, 2013:437-442. [6]S. Hochreiter, J. Schmidhuber. Long short-term memory. Neural Computation, 1997, 9(8):1735-1780. [7]A. G. Howard, M. Zhu, B. Chen, et al. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. [8]M. Sandler, A. Howard, M. Zhu, et al. MobileNetV2: Inverted residuals and linear bottlenecks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, 2018:4510-4520. [9]B. Jacob, S. Kligys, B. Chen, et al. Quantization and training of neural networks for efficient integer-arithmetic-only inference. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, 2018:2704-2713. [10] D. Li, C. Rodriguez, X. Yu, et al. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), Snowmass, 2020:1448-1458
Copyright @ 2020-2035 STEMM Institute Press All Rights Reserved