Transformer-based Multimodal Fusion for Spatiotemporal Synchronization of Rehabilitation Signals: A Precise Identification Method for Unilateral Compensatory Movements

DOI: https://doi.org/10.62517/jmhs.202605230

Author(s)

Jiayi Shen, Luyan Shen, Sitong Ruan, Zhihui Xu, Jingyi Xu*

Affiliation(s)

Artificial Intelligence College, Zhejiang Dongfang Polytechnic, Wenzhou, Zhejiang, China *Corresponding Author

Abstract

Unilateral compensatory movements are prevalent abnormal motor patterns in stroke patients during rehabilitation, impairing neural plasticity training and causing secondary injuries. Conventional multimodal rehabilitation systems are plagued by poor signal spatiotemporal synchronization, failure in dynamic weighting of critical information and high misdiagnosis rates of compensatory motions. This paper develops a Transformer-based multimodal fusion framework. A hardware-software co-synchronization strategy realizes millisecond-level alignment of EEG, sEMG and IMU signals. The cross-modal attention fusion model mines spatiotemporal correlations between neuromuscular and kinematic data, while a fine-grained classifier identifies six upper-limb compensatory patterns. Tests on 20 stroke patients yield a misjudgment rate of only 4.2%, far outperforming traditional algorithms. Clinically, the system delivers real-time reminders to cut compensatory movements and boost rehabilitation efficiency.

Keywords

Transformer; Multimodal Fusion; Spatiotemporal Synchronization; Unilateral Compensatory Movements; Stroke Rehabilitation; Real-Time Intervention

References

[1] Zhang X, Wang Y, Jin J, et al. A lightweight CNN-based EEG classification model for edge computing. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 1234-1243. [2] Li Y, Wang H, Zhang L, et al. EdgeBCI: An edge computing framework for low-latency brain-computer interfaces. IEEE Internet of Things Journal, 2024, 11(8): 13456-13467. [3] Gao S, Chen J, Chen X, et al. Temporal dynamics and physical prior multimodal network for rehabilitation physical training evaluation. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2024, 32: 1897-1908. [4] Chen X, Wang L, Liu C. Application of multithreaded asynchronous scheduling in real-time brain-computer interface systems. Computer Engineering and Applications, 2023, 59(12): 189-195. [5] Zhao J, Rao Y. Adaptive Multimodal Temporal Transformer for Bio Signal Driven Stroke Rehabilitation Prognosis. bioRxiv, 2025: 2025.10.18.683234. [6] Leeb R, Friedman D, Müller-Putz G R, et al. Towards brain-computer interface driven virtual reality systems: Design and implementation. Presence: Teleoperators and Virtual Environments, 2007, 16(3): 290-307. [7] Wolpaw J R, Birbaumer N, McFarland D J, et al. Brain-computer interfaces for communication and control. Clinical Neurophysiology, 2002, 113(6): 767-791. [8] Shadmehr R, Smith M A, Krakauer J W. Error correction, sensory prediction, and adaptation in motor control. Annual Review of Neuroscience, 2010, 33: 89-108. [9] Xu G H, Zhang J, Wang J. Advances in the application of brain-computer interface technology in stroke rehabilitation. Chinese Journal of Rehabilitation Medicine, 2023, 38(2): 267-272. [10] Wang X, Feng Y, Li Y, et al. Technology-Based Compensation Assessment and Detection of Upper Extremity Activities of Stroke Survivors: Systematic Review. Journal of Neuroengineering and Rehabilitation, 2022, 19(1): 78.