i3DTI: Interactive 3D Tele-Immersion

3D Tele-Immersion (3DTI) approaches offer collaborative augmented virtuality, by immersing multiple geographically distributed users in a single virtual world. Multiple cameras are used to capture remote users and reconstruct their 3D models, to be rendered in the virtual world. Large volumes of noisy data, captured by the cameras, are cleaned, processed, transmitted, and rendered every frame. The complexities in processing and large transmission payloads result in high latency, low frame rate rendering, leading to very limited interactions between the users of the system. Interactive 3D Tele-Immersion (i3DTI) systems allow geographically distributed users to be immersed in highly engaging and interactive virtual worlds.

This dissertation presents a set of novel approaches, that improve the quality and performance of all of the stages of an i3DTI application. An image based meshing approach reduces the time taken for the 3D reconstruction of the captured data, to less than a millisecond. To ensure low latency even over the internet, a skeleton based prediction strategy is presented, that reduces the quantity of data transmitted per frame to just a few hundred bytes, while still maintaining good quality rendering. Naturalistic full body interactions, based on the skeleton that is estimated using multiple RGB-D cameras, keeps the users engaged while using the system. Current 3DTI systems use internal clocks to measure the latency experienced by the users, while ignoring the time for capture, rendering, screen refreshing, etc. Two novel, millisecond accurate approaches, for measuring the latency felt by the user, are presented. One of the approaches measures the latency across the internet while the system is not actively in use, and the other measures the local latency while the system is in use. All of the approaches are implemented as a framework in a highly scalable, performance optimized, easy to use, and extendable architecture. Multiple applications, catering to domains from Tele-Medicine to education, were created using the framework. The patient trials of a remote patient diagnosis system, that was implemented between two geographically distributed locations using the i3DTI framework, was highly appreciated by the users; and the remote patient diagnosis was highly correlated to the in-person diagnosis.