An Adaptable VR Software Framework for Collaborative Multi-modal Haptic and Immersive Visual Display

doi:10.16182/j.issn1004731x.joss.19-VR0440

Abstract

Abstract: The spatiotemporal consistency of multi-modal haptic and immersive visual feedback, as well as the integration of hardware devices in the existing VR software framework, restricts the rapid development of visuo-haptic fusion scenarios in different industries such as medical, business, and entertainment. In order to tackle this problem, an adaptable VR software framework combining multi-modal haptic and immersive visual display is proposed. To meet the different requirements of visual rendering, haptic rendering and hardware control, the double-layer and three-layer architectures are devised according to the different precision requirements of visuo-haptic fusion feedback. Experimental results show that the framework has the characteristics of supporting the synchronous feedback of visuo-haptic fusion, the convenient integration of diverse haptic devices, the simplicity of scene development and the rapid replacement of interactive scene, and it can meet the requirements of different application fields.

Key words: virtual reality, multi-modal haptic, human-machine interaction, visuo-haptic fusion, adaptable software framework

CLC Number:

TP391.9

Guo Yuan, Tong Qianqian, Zheng Yukai, Wang Ziqi, Zhang Yuru, Wang Dangxiao. An Adaptable VR Software Framework for Collaborative Multi-modal Haptic and Immersive Visual Display[J]. Journal of System Simulation, 2020, 32(7): 1385-1392.

References 17

[1]	Fukumoto M, Sugimura T.Active click: tactile feedback for touch panels[C]// CHI'01 Extended Abstracts on Human Factors in Computing Systems. New York, USA: ACM, 2001: 121-122.
[2]	Yatani K, Truong K N.SemFeel: a user interface with semantic tactile feedback for mobile touch-screen devices[C]// Proceedings of the 22nd annual ACM symposium on User interface software and technology. New York, USA: ACM, 2009: 111-120.
[3]	Sawada H, Takeda Y.Tactile pen for presenting texture sensation from touch screen[C]// 2015 8th International Conference on Human System Interaction (HSI). Warsaw, Poland: IEEE, 2015: 334-339.
[4]	Yang G H, Jin M, Jin Y, et al.T-mobile: Vibrotactile display pad with spatial and directional information for hand-held device[C]// 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems. Taipei, China: IEEE, 2010: 5245-5250.
[5]	Sun Z, Miao X, Li X.Design of a bidirectional force feedback dataglove based on pneumatic artificial muscles[C]// 2009 International Conference on Mechatronics and Automation. Changchun, China: IEEE, 2009: 1767-1771.
[6]	Cassar D J, Saliba M A.A force feedback glove based on magnetorheological fluid: preliminary design issues[C]// Melecon 2010-2010 15th IEEE Mediterranean Electrotechnical Conference. Valletta, Malta: IEEE, 2010: 618-623.
[7]	Gu X, Zhang Y, Sun W, et al.Dexmo: An inexpensive and lightweight mechanical exoskeleton for motion capture and force feedback in VR[C]// Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. New York, USA: ACM, 2016: 1991-1995.
[8]	Choi I, Hawkes E W, Christensen D L, et al.Wolverine: A wearable haptic interface for grasping in virtual reality[C]// 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Daejeon, South Korea: IEEE, 2016: 986-993.
[9]	Wang D, Ohnishi K, Xu W.Multimodal Haptic Display for Virtual Reality: A Survey[J]. IEEE Transactions on Industrial Electronics (S0278-0046), 2020, 67(1): 610-623.
[10]	Mansour N A, El-Bab A M R F, Abdellatif M. Design of a novel multi-modal tactile display device for biomedical applications[C]// 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob). Rome, Italy: IEEE, 2012: 183-188.
[11]	Stanley A A, Okamura A M.Deformable model-based methods for shape control of a haptic jamming surface[J]. IEEE transactions on visualization and computer graphics (S1077-2626), 2016, 23(2): 1029-1041.
[12]	Murakami T, Person T, Fernando C L, et al.Altered touch: miniature haptic display with force, thermal and tactile feedback for augmented haptics[C]// ACM SIGGRAPH 2017 Emerging Technologies. New York, USA: ACM, 2017.
[13]	Choi I, Culbertson H, Miller M R, et al.Grabity: A wearable haptic interface for simulating weight and grasping in virtual reality[C]// Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. New York, USA: ACM, 2017: 119-130.
[14]	Choi I, Ofek E, Benko H, et al.Claw: A multifunctional handheld haptic controller for grasping, touching, and triggering in virtual reality[C]// Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. New York, USA: ACM, 2018.
[15]	Otaduy M A, Lin M C.Sensation preserving simplification forhaptic rendering[J]. ACM Transactions on Graphics (S0730-0301), 2003, 22(3): 543-553.
[16]	Wang D, Zhang X, Zhang Y, et al.Configuration-based optimization for six degree-of-freedom haptic rendering for fine manipulation[J]. IEEE Transactions on Haptics (S1939-1412), 2013, 6(2): 167-180.
[17]	Wang D, Shi Y, Liu S, et al.Haptic simulation of organ deformation and hybrid contacts in dental operations[J]. IEEE Transactions on Haptics (S1939-1412), 2014, 7(1): 48-60.