Design and Wizard-of-Oz Evaluation of a Tri-Modal Contactless In-Vehicle HMI for High-Noise and High-Vibration Cabins_Vol. 4 No. 2 (JIKE 2026)_Journal of Intelligence and Knowledge Engineering (ISSN: 2959-0620)

Home > Journal of Intelligence and Knowledge Engineering (ISSN: 2959-0620) > Vol. 4 No. 2 (JIKE 2026) >

Design and Wizard-of-Oz Evaluation of a Tri-Modal Contactless In-Vehicle HMI for High-Noise and High-Vibration Cabins

DOI: https://doi.org/10.62517/jike.202604230

Author(s)

Zhenghao Yu

Affiliation(s)

Department of Eniegering, Faculty of Electronic and Electrical Engineering, The University of Hong Kong, Hong Kong, China *Corresponding author.

Abstract

Harsh cabins in mining, construction, and defense degrade touch human–machine inter- faces for secondary tasks, while single-modality contactless input stays fragile under coupled noise and vibration. A sequenced tri-modal protocol with dwell locking, head confirma- tion or cancellation, and optional voice refinement is specified and evaluated in a desktop Wizard-of-Oz browser under parameterized disturbance. An operator survey (N = 48) fixed timing constants; N = 30 participants compared touch (A), pooled single-modality contact- less (B), and tri-modal (C) under a fixed Hard preset. Holm–Bonferroni contrasts show C reduced error rate and NASA–TLX versus B (p <.01), shortened completion time versus B (p <.05), and improved SUS versus B, while A remained fastest with the lowest descriptive errors. Conclusions are interaction-level evidence under simulation, not ISO-calibrated cabin equivalence or production gaze or speech benchmarks.

Keywords

Multimodal Interaction; In-Vehicle HMI; Special-Purpose Vehicles; High-Noise Strong-Vibration; Gaze Dwell; Head Gesture; Voice Refinement; Wizard-of-Oz.

References

[1] Lorenz, S., Helmert, J.R., Anders, R., Wölfel, C., Krzywinski, J. (2020) UUX evaluation of a digitally advanced human–machine interface for excavators. Multimodal Technologies and Interaction, 4(3): 57. [2] Tan, Z., Dai, N., Su, Y., et al. (2022) Human–machine interaction in intelligent and con- nected vehicles: A review of status quo, issues, and opportunities. IEEE Transactions on Intelligent Transportation Systems, 23(9): 13954–13975. [3] Murali, P.K., Kaboli, M., Dahiya, R. (2022) Intelligent in-vehicle interaction technologies. Advanced Intelligent Systems, 4(2): 2100122. [4] Khan, M.Q., Lee, S. (2019) Gaze and eye tracking: Techniques and applications in ADAS. Sensors, 19(24): 5540. [5] Dua, M., Akanksha, Dua, S. (2023) Noise robust automatic speech recognition: Review and analysis. International Journal of Speech Technology, 26: 475–519. [6] Zimmermann, M., Bengler, K. (2013) A multimodal interaction concept for cooperative driv- ing. In: 2013 IEEE Intelligent Vehicles Symposium (IV). IEEE, Piscataway, NJ. pp. 1285– 1290. [7] Nesselrath, R., Moniri, M.M., Feld, M. (2016) Combining speech, gaze, and micro-gestures for the multimodal control of in-car functions. In: Proceedings of the 12th International Conference on Intelligent Environments (IE). IEEE, Piscataway, NJ. pp. 190–193. [8] Aftab, A.R. (2019) Multimodal driver interaction with gesture, gaze and speech. In: Pro- ceedings of the 21st ACM International Conference on Multimodal Interaction (ICMI ’19). ACM, New York. pp. 487–492. [9] Ayoub, J., Zhou, F., Bao, S., et al. (2019) From manual driving to automated driving: A review of 10 years of AutoUI. In: Adjunct Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, New York. pp. 70–90. [10] Dahlbäck, N., Jönsson, A., Ahrenberg, L. (1993) Wizard of Oz studies—why and how. Knowledge-Based Systems, 6(4): 258–266. [11] Brooke, J. (1996) SUS: A “quick and dirty” usability scale. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, A.L. (Eds.), Usability Evaluation in Industry. Taylor & Francis, London. pp. 189–194. [12] Ojsteršek, T.C., Topolšek, D. (2019) Eye tracking use in researching driver distraction: A sci- entometric and qualitative literature review approach. Journal of Eye Movement Research, 12(3): Article 5. [13] Roider, F., Rümelin, S., Pfleging, B., Gross, T. (2017) The effects of situational demands on gaze, speech and gesture input in the vehicle. In: Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutoUI). ACM, New York. pp. 94–102. [14] Khamis, M., Alt, F., Bulling, A. (2018) The past, present, and future of gaze-enabled handheld mobile devices. In: Proceedings of the 20th International Conference on Human– Computer Interaction with Mobile Devices and Services (MobileHCI ’18). ACM, New York. pp. 1–17. [15] Mohan, P., Goh, W.B., Fu, C.-W., Yeung, S.-K. (2018) DualGaze: Addressing the Midas touch problem in gaze mediated VR interaction. In: 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, Piscataway, NJ. pp. 79–84. [16] Prabhakar, G., Ramakrishnan, A., Murthy, L.R.D., et al. (2020) Interactive gaze and fin- ger controlled HUD for cars. Journal on Multimodal User Interfaces, 14: 101–121. [17] Martelaro, N., Ju, W. (2017) WoZ Way: Enabling real-time remote interaction proto- typing. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooper- ative Work and Social Computing (CSCW ’17). ACM, New York. pp. 169–182. [18] Detjen, H., Pfleging, B., Schneegass, S. (2020) A Wizard of Oz field study to understand non- driving-related activities, trust, and acceptance of automated vehicles. In: Proceedings of the 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutoUI). ACM, New York. pp. 19–29. [19] Schlögl, S., Doherty, G., Luz, S. (2024) Wizard of Oz experimentation for language technol- ogy applications: Challenges and tools. arXiv :2402.14563. [20] Hart, S.G., Staveland, L.E. (1988) Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (Eds.), Human Mental Workload. North-Holland, Amsterdam. pp. 139–183. [21] Detjen, H., Geisler, S., Schneegass, S. (2020) Maneuver-based control interventions during automated driving: Comparing touch, voice, and mid-air gestures as input modalities. In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, Piscataway, NJ. pp. 3268–3274. [22] National Institute for Occupational Safety and Health (NIOSH). (1998) Criteria for a Recommended Standard: Occupational Noise Exposure. DHHS (NIOSH) Publication No. 98-126. U.S. Department of Health and Human Services, Cincinnati, OH. Available: https://www.cdc.gov/niosh/docs/98-126/ [23] Hussain, J., Ul Hassan, A., Muhammad Bilal, H.S., et al. (2018) Model-based adaptive user interface based on context and user experience evaluation. Journal on Multimodal User Interfaces, 12(1): 1–16. [24] Detjen, H., Geisler, S., Schneegass, S. (2021) Driving as side task: Exploring intuitive input modalities for multitasking in automated vehicles. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York. Article 180, pp. 1–6.