Google has made some neat tech that makes VR headsets "transparent" for MR use.
Google has made some neat technology that makes VR headsets "transparent" for Mixed Reality use.
According to a recent post from the Google VR blog Daydream Labs and Google Research teamed up to start exploring how to solve for the problem of seeing facial expressions when watching a person use VR without an avatar representation.
Above is artist Steve Teeple in Tilt Brush, shown in traditional mixed reality on the left and with headset removal on the right, which reveals the face and eyes for a more engaging experience.
According to Avneesh Sud- SOFTWARE ENGINEER at GOOGLE RESEARCH
"The first step to removing the VR headset is to construct a dynamic 3D model of the person’s face, capturing facial variations as they blink or look in different directions. This model allows us to mimic where the person is looking, even though it's hidden under the headset.
Next, we use an HTC Vive, modified by SMI to include eye-tracking, to capture the person’s eye-gaze from inside the headset. From there, we create the illusion of the person’s face by aligning and blending the 3D face model with a camera’s video stream. A translucent "scuba mask" look helps avoid an "uncanny valley" effect.
Finally, we composite the person into the virtual world, which requires calibrating between the Vive tracking system and the external camera. We’re able to automate this and make it highly accurate so movement looks natural. The end result is a complete view of both the virtual world and the person in it, including their entire face and where they’re looking."
