Quantifying Experience and Task Performance in 3D Serious Games
Abstract
Abstract
Mixed reality systems allow the development of different 3D immersive games, by immersing
a live captured 3D model of a person in a virtual environment and enabling interactions and
collaborations among geographically distributed people. Serious gaming is one application
domain for a mixed reality system in which games are developed for primary purpose of
education or training rather than just entertainment. Different aspects of a serious game
such as visual, interaction, immersion, etc. influence the user’s perceived experience. Task
performance in a serious game reflects how efficiently and accurately users carry out the
assigned tasks. For instance, in a serious game for a virtual STEM (Science, Technology,
Engineering and Mathematics) experiment, the user’s task performance can be characterized
by how accurately the user follows the given procedure and how efficiently the goals are
accomplished. User’s task performance in such serious games is typically influenced by
his/her experience of the provided virtual environment for carrying out the assigned task. In
this dissertation, we focus on the problem of quantifying experience and task performance
in 3D serious games by addressing the following questions - (i) Can we, and if so how do we,
quantify the user’s experience and potentially improve it in different serious games? (ii) Can
we, and if so how do we, map the user’s task performance in a real world scenario to the
corresponding virtual world serious game? Also, is there a correlation that exists between
the user’s task performance and experience in serious games?
Since serious games is a wide domain covering countless applications, it would be difficult
if not impossible to generalize and answer the above questions for all of them. Hence, we
focus on solving the research questions for 3 different domains of serious games developed
using a mixed reality framework, namely Exergames, Multi-Modal Collaborative Virtual
Laboratory (MMCVL) and Penalty training game. In order to quantify the visual experience,
a learning-based objective measure is developed that emulates human perception of the 3D
human open mesh quality. For a mixed reality application, a large amount of 3D data is
generated and transmitted across the network, even for a single RGB-D Kinect camera. To
reduce this data, based on the available bandwidth, a visual quality based vertex selection
technique and a sweep-line based meshing technique is used. The user experience of a
mixed reality game is improved, if accurate interactions are provided for a wide range of
motion. Multiple cameras are needed to provide a complete representation of the person
from all directions. A fast skeleton-based re-calibration method is developed that performs
continuous and simultaneous extrinsic calibration of multiple Kinect cameras. Skeletal poses
from multiple Kinect cameras are combined in order to generate a high quality combined 3D
point cloud model. User studies are performed to evaluate the effect of 3 aspects - visual,
interaction and immersion, on the overall quality of experience. For automatically assessing
the task performance of a single user or a group of users in a 3D serious game, we formally
express the assessment logic using an Augmented Hierarchical Task Network (A-HTN). Game
authors are provided with an authoring script to help them incorporate the assessment logic
in the game. Recording mechanism is used to store user’s task performance for assessment
as well as for future reference. User studies are conducted on the 3 serious games domains
and correlation analysis is performed to show that the user’s task performance improves if
high quality of experience is provided in 3D serious games.