What if you could walk through a film?

My work addresses a novel technique for free navigation in a Virtual Reality film, and explores the spatial - temporal narrative implications of such a technique.

With the increasing popularization of VR filmmaking, under current technology, filmmakers who choose to represent a realistic scene are faced with a choice between either graphically intensive real-time 3D rendering on a high-end VR system, or a solution based on Omnidirectional Stereoscopic (ODS) Imaging captured by a camera. Commonly known as 360 videos, they suffer from a fixed viewpoint.

To address this specific issue, many recent advancements in the field proposed novel view reconstruction from spherical panoramas, planar image - based rendering, light field rendering, etc. for free-viewpoint navigation. 

I explored the interactive implications of such techniques when they are implemented for VR filmmaking. Specifically, the temporal relationship of viewpoints and the experience of navigating through them.

I moved an Omnidirectional Stereoscopic camera continuously through the scene. During playback, the system does not display frames sequentially in a traditional manner. Instead, as the viewer moves through the scene, the system displays frames based on the realtime tracked position of the headset.

I have found that while the technical package is complicated to explain to a non-exepert, the experience is delightfully intuitive. 

Movement controls time. Forward movement advances time, backward movement rewinds it, and standing still slows time until it freezes. The viewer is essentially walking through space and time at once.

From an initial proof of concept with a linear path, walking down the famous Shibuya crossing in Tokyo, I continued to develop the technique through installations and experimented with an infinitely looping circular path around NYC grand central terminal’s clock tower.

I aim to integrate multiple optimizations and techniques into an easy to use open source package for filmmakers. Including storage optimizations, novel view synthesis, and locomotion redirection - in the case of the grand central installation, my real-time redirection implementation shrunk a 60ft diameter film scene into a 20ft diameter installation with no noticeable discomfort. 

Currently, I am exploring the full narrative potential of the technique and other potential spatial topologies beyond a line or a circle. 

Unique to each medium of storytelling, there is a defining technique that elevates it towards a form of art - a story, experience, or memory only this medium can deliver. Just like what Eisenstein’s theory of Montage meant to the development of cinema as a unique art form , Pacing Time is a technique that explores the full potential of VR cinema, and unlocks a new way to tell a story across space and time. I can’t wait to see what stories can be told in the future.

The interactive installation was developed and showcased at Yale CCAM in December 2021.