Random thoughts on filmmaking, life and science. Subscribe via email.

Virtual Reality Cinema? Creating a Narrative Artificial Intelligence for VR

First things first, Virtual Reality cinema is a misnomer.

The reason being, in VR you create an experience and not just tell a story. Therefore, the language of cinema is simply obsolete.

Filmmakers are having a hard time coping with the uncertainty and lack of control, but they are learning quickly. This article pushes the bar a bit further by defining a Narrative AI (artificial intelligence) strategy for cinematic Virtual Reality.

The four types of VR

The four types of cinematic VR are mapped in the image above.   (Credit   Devon Dolan   and   Kent Bye  )

The four types of cinematic VR are mapped in the image above.
(Credit Devon Dolan and Kent Bye)

In “Ghost”, the viewer is an unseen observer.

By contrast, “Character” is where the viewer is an actual participant in the drama – thus allowing interaction in the world.

Out of the four types of VR, 360 video is the least common denominator. It’s so low in fact that most VR purists don’t even consider it VR at all. In 360 video, you can move your head in a shot and see all around which is way cool but it’s entry level VR. The real goal of VR is to put the viewer in the scene as a participant who’s able to influence the story.

This is represented by the “Character with Impact” box in the image above and it’s the highest form of VR.

Later this year, VR penetration will start to accelerate and a thirst for quality content will follow in years to come. It’s now possible to create next generation VR … here’s how.

Goal #1: Create a World

In real life we create stories out of experiences we’ve had with people and environments.

In VR the same is true so the goal is to create a world where this is ripe to happen. The central questions are:

  • What type of world will give the viewer a rewarding experience?
  • What type of experience do I want them to have?
  • What characters inhabit the world?
  • What are the character backstories?
  • Where will these characters take the viewer and what emotions will they experience as a result?

At face value, these questions appear similar to questions you ask at the beginning of any story outline but the difference is that the emphasis is on experience.

VR is inherently non-linear which demands experience-based storytelling over linear storytelling. In experience-based storytelling an emergent narrative reigns over a linear based narrative and that starts with the world.

Goal #2: Enhance Emotion through Immersion

Many psychological studies have been conducted on VR’s ability to enhance emotional connection to material.

The sheer fact of putting on the headset allows for far greater emotional responses than we get from cinema. However, the key to unlock VR’s potential relies on the creators ability to maximize the viewers immersion in the world. This is another argument for aiming for the “Character with Impact” square in the image above.

Allowing the viewer to become an active participant who impacts the story is the key to immersion maximization. The research shows a direct correlation between immersion and emotional impact.

Goal #3: Create a Narrative Artificial Intelligence

This is the secret sauce of immersion. This is how you make the viewer a participant that affects the story – through collect nodes and story modes.

The Collect Node

Collect nodes learn about the viewer by collecting inputs from the user in order for the story nodes to later make decisions to maximize immersion.

They collect tactile input (controller, or movement), audio input (speech, laughter), and visual information from the head mounted display (HMD). The collect nodes are placed at strategic places in the world in order to learn the greatest about the viewer.

For instance, if you’re making a comedy and you’ve placed a comedic character in the world you may want to test if the viewer is engaged and also if they’re laughing. Collect nodes can get this type of information.

The Story Node

Story nodes change the experience for the viewer based on the inputs collected from collect nodes.

To continue with the previous example, if we’ve collected information from the viewer that suggests that they like a particular type of comedy (via laughter and attention … i.e visual input) then the story node may branch the experience in a way to highlight this character and this type of comedy.

While in graduate film school at USC, I made a film that performed this task in a simplistic form. It was a comedic film that listened for laughter and changed the story if laughter was detected or not. It was a test on whether a particular form of comedy (physical, slapstick etc.) worked for the viewer. It was developed in Flash using filmed elements and actionscript. The project was ahead of its time but it was useful as a precursor to the theory presented here.

Narrative AI Experience Flow

The above narrative AI flow shows how collect nodes and story nodes work together to shape the experience over time. The flow appears linear because I put time on the X axis but the flow is really a non-linear experience. Story nodes are allowed to loop back and across into others – thereby creating a web of interactivity and possibility.

It’s a new medium so take the challenge and think about how to tell stories in a new way. Take 360 video to the next level by adding a narrative AI!