Disney Research has leveraged big data to predict soccer movements, improve sports broadcasts by having automated cameras learn from human operators and develop a “smart search” for sports plays.
Work in the digital realm had only indirectly influenced the human world. Historical soccer data later informed attack strategies, and data from human camera operators allowed researchers to teach automated systems.
But now, Disney Research has achieved a machine-aided task in real-time.
Using a motion capture system, Günter Niemeyer and Matthew K.X.J. Pan rendered a series of visual cues that allow an individual wearing a VR headset to catch a tennis ball in real life.
“The idea is really quite simple,” Pan said. “We essentially track a few items, and by track I mean we look at the 3D position and orientation of several objects in a space. That includes the hands, the head as tracked by the head-mounted display, and the ball. So all those things are tracked and we render each of those objects in a 3D environment.”
Once Niemeyer and Pan successfully tracked the ball in a 3D environment, Pan generated three “assistances” — visualizations of the ball itself, the ball’s predicted trajectory and a ‘target catching point’ that shows where the ball will ultimately land — to assist the user in catching the ball.
Get The Latest Sports Tech News In Your Inbox!
“To generate the predictive visualizations we just use simple high-school physics pretty much,” Pan said. “So you know those equations for like F = MA and gravity affects objects? It’s essentially those equations which are driving the prediction and the generation of the parabolic trajectory and an end target.
“So using those equations we can come up with the time it takes for the object to reach a specific target in the future as it’s undergoing parabolic motion and the place that it will arrive at, with a fair amount of accuracy.”
In a series of tests using different combinations of the three aforementioned assistances (the rendered ball, the rendered predicted trajectory of the ball and the rendered target catching point lying on the predicted trajectory of the ball) the user caught the ball 132/140 times.
“To me, all of this is to me really interesting because we are so used to living in a world of these beautiful virtual images but we deep down know they’re all virtual,” Niemeyer said. “It’s always in the back of your mind. You know it’s a different, virtual thing that isn’t real. And when something suddenly hits you, and even though you saw it in VR coming at you, your brain goes, ‘Whoa, wait. It’s real!’ It has that moment where all of a sudden it kind of does come to life and it is pretty amazing.”
“By having that haptic feeling of catching something while you’re seeing that you’re catching something, it’s just a really, really cool feeling,” Pan added. “And I don’t know how to best describe it. It’s kind of freaky in a way.”
So what comes next for Niemeyer and Pan?
“In some sense what we do is we try to figure out what are kind of new, novel technologies, what problems that people have with them can we solve them and to really just build up a stable of cool, interesting, novel things that then maybe sooner or later become something or not,” Niemeyer said. “So there’s definitely an element of exploration in this, and so at the moment we don’t have any firm plans to use this for anything in particular.”
Indeed, Pan is still conducting user studies and analyzing different environments and visualization aids (“assistances”), but “Catching a Real Ball in Virtual Reality” is first and foremost a research project.
That doesn’t mean the implications of Niemeyer and Pan’s success is confined to the research setting, though.
The prospect of combining virtual and physical dynamic interactions to enrich virtual reality experiences has widespread commercial implications, the most impressive of which could allow users to further engage in a virtual, sports world, like Wimbledon.
“If you’re able to track a tennis court or track objects within a tennis-court sized space, (it could) definitely (work),” Niemeyer said. “You could have each of the tennis players put on a VR headset, you could still have the prediction algorithms running because the ball is still following projectile motion. And yeah, you could totally change the background…you could totally have a setting where it’s like Wimbledon or like if you were playing basketball at the center of the (NBA) All-Star game.”