Teleoperation over 5G Networks: Enabling Haptic Interaction in Mobile Audio-Visual Communications
Final Report Abstract
While audiovisual information provides a user with a satisfactory impression of being present in a remote environment, physical interaction and manipulation are not supported. True immersion of being present in a remote environment requires the ability of physical interaction with objects or people. Touching and manipulating objects remotely becomes possible if we augment traditional audiovisual communications by the haptic modality. Haptic communication is a relatively young field of research that has the potential to revolutionize human-human and human-machine interaction. Recently, 5G mobile networks have gained considerable research attention, including the idea of a control-oriented communication solution (also called the Tactile Internet). The targeted low latency communication, high level of coverage and reliability and improved throughput make 5G networks very suitable for teleoperation applications with haptic feedback. From the application perspective, the teleoperation performance is highly affected by the adopted haptic communication schemes and the QoS provided by the underlying 5G communication network. So far, however, there is neither a common understanding about the preferred haptic communication approaches under different QoS conditions, nor generalizable results about the required QoS to achieve a certain teleoperation quality. In this project, we have developed a number of solutions for time-delayed teleoperation over communication networks, which combine different haptic data reduction approaches with different stability-ensuring control schemes. After studying how these combinations affect teleoperation performance for various QoS and task requirements, we have proposed the subjectively best possible teleoperation quality as a function of different control schemes and network QoS supports. According to the proposed quality evaluation results, we have further developed a resource allocation scheme to gain the maximum overall QoE for teleoperation sessions sharing the same communication network. In addition, our contributions to the IEEE P1918.1.1 kinesthetic codec standard is also one of the major outcomes from this project. Several challenges were faced during the execution of this project. While combining different control schemes with our previously proposed kinesthetic data reduction approach, we observed that the data reduction leads to strong conservatism and results in sugnificant quality degradation. It is even more challenging when the packet loss rate is high and the communication delay is not constant. With proper knowledge about the network parameters, this artifact can be mitigated by adopting energy compensation schemes, but cannot be fully eliminated. Meanwhile, we have encountered some practical issues while implementing the proposed approaches on real teleoperation systems. The design of the control and compensation schemes must be adaptive to different devices. We see two major future applications for the outcome of this project. First, we envision a growing use of teleoperation in future life for skill training, tele-medical applications, teleoperated driving, tele-service robot, etc. The proposed haptic control and communication schemes can be applied to address the core challenges when running teleoperation systems over public networks. Secondly, the proposed haptic codec standard as well as the reference software provide a “plugand-play teleoperation system” solution and testbed. This is particularly interesting for researchers targeting further extensions and improvements of the state-of-the-art teleoperation technologies.
Publications
- “QoE-Driven Uplink Scheduling for Haptic Communications over 5G Enabled Tactile Internet,” IEEE International Symposium on Haptic Audio-Visual Environments and Games, Dalian, China, September 2018
Liu, Siwen; Li,Molin; Xu, Xiao; Liu, Qian; Steinbach, Eckehard
(See online at https://doi.org/10.1109/HAVE.2018.8547503) - “Toward Haptic Communications Over the 5G Tactile Internet,” IEEE Communications Surveys & Tutorials 20 (4), 2018, 3034-3059
Antonakoglou, Konstantinos; Xu, Xiao; Steinbach, Eckehard; Mahmoodi, Toktam; Dohler, Mischa
(See online at https://doi.org/10.1109/COMST.2018.2851452) - “Elimination of Cross-dimensional Artifacts in the Multi-Dof Time Domain Passivity Approach for Time-delayed Teleoperation with Haptic Feedback,” IEEE World Haptics Conference, Tokyo, Japan, July 2019 [best paper award]
Xiao Xu, E. Steinbach
(See online at https://doi.org/10.1109/WHC.2019.8816074) - “Haptic Codecs for the Tactile Internet,” Proceedings of the IEEE 107 (2), 2019, 447-470
Steinbach, Eckehard; Strese, Matti; Eid, Mohamad; Liu, Xun; Bhardwaj, Amit; Liu, Qian; Al-Ja'Afreh, Mohammad; Mahmoodi, Toktam; Hassen, Rania; El Saddik, Abdulmotaleb; Holland, Oliver
(See online at https://doi.org/10.1109/JPROC.2018.2867835) - “Haptic Data Reduction for Timedelayed Teleoperation Using the Input-to-State Stability Approach,” IEEE World Haptics Conference, Tokyo, Japan, July 2019
Xu, Xiao; Liu, Qian; Steinbach, Eckehard
(See online at https://doi.org/10.1109/WHC.2019.8816106) - “Adaptive Packet Rate Control for the Mitigation of Bursty Haptic Traffic in Teleoperation Systems,” IEEE Haptics Symposium, Washington, DC, March 2020
Gui, Ming; Xu, Xiao; Steinbach, Eckehard
(See online at https://doi.org/10.1109/HAPTICS45997.2020.ras.HAP20.18.6a51e1e1) - “Integrating Haptic Data Reduction with Energy Reflection Based Passivity Control for Time-delayed Teleoperation,” IEEE Haptics Symposium, Washington, DC, March 2020
Xu, Xiao; Panzirsch, Michael; Liu, Qian; Steinbach, Eckehard
(See online at https://doi.org/10.1109/HAPTICS45997.2020.ras.HAP20.2.4ff61dc8) - “On the Quality-of-Learning for Haptic Teleoperation-based Skill Transfer over the Tactile Internet,” IEEE GLOBECOM, Taipei, Taiwan, December 2020
Gülecyüz, Basak; Xu, Xiao; Noll, Andreas; Steinbach, Eckehard
(See online at https://doi.org/10.1109/GLOBECOM42002.2020.9322290) - A Novel Energy Compensation Scheme for Quality Enhancement in Time-delayed Teleoperation with Multi-DoF Haptic Data Reduction and Communication,” IEEE Transactions on Haptics
X. Xu, H. Singh, Q. Liu, M. Panzirsch, T. Hulin, and E. Steinbach
(See online at https://doi.org/10.1109/TOH.2021.3103379)
