Project Group "Virtual Self": A Mixed Reality Experience

Goals:

Current Virtual Reality applications lack a representation of the user's body: you cannot see your arms if you extent your hands, and you cannot see your legs and feet if you look down. More importantly, your body cannot interact with the virtual world: you cannot kick a virtual ball, for example, since the application knows nothing about your feet.

In this project group, we will bring a model of the user's body into the virtual world so that the user can see and use it. First experiments suggest that this model does not have to be realistic to provide a sense of presence in the virtual world; it can be a symbolic representation as long as movements are reproduced faithfully and without latency.

We will use the HTC Vive head-mounted display to provide the virtual world. We plan to derive the body model from two different sensor systems:

• the ART optical tracking system with a set of body tracking targets (arms, hands, waist, legs, feet) for the main body tracking, and
• the Leap Motion device for hand and finger tracking.

The information of both systems will be fused into a single skeleton-based model of the user's body. This model will then be used for two purposes: first, it will be rendered into the virtual scene so that the user can see it, and second, it will be used to allow interactions with the virtual world (for example kicking or picking up a virtual ball).

This virtual self experience will be showcased with a demo application that the project group will design and implement. For example, virtual objects of different kinds might fly towards the user who needs to pick them up, kick them away or sort them into buckets using his own body instead of some virtual tool represented by a VR controller.

To achieve these goals, the project group will have the following four main work packages:

1. Design and implement a skeleton-based body model based on the information from the ART and Leap Motion trackers.
2. Design and render a visual representation of the body model so that the user can see himself.
3. Design and implement a representation of the body model suitable for physically-based interaction with the virtual world.
4. Design and implement an example application suitable to demonstrate this new virtual self experience.

Participating in this project group requires team spirit, perseverance, and an interest in Computer Graphics. You need to have passed Computer Graphics I. Additionally, it is helpful to have participated in at least one other computer graphics course, e.g. Virtual Reality, Computer Graphics III, or Computer Graphics Practical Training.

If you are interested or have any questions, please contact Martin Lambers.