Interactive storytelling

The global sales of virtual reality goggles is predicted to reach twelve million units by 2016. The entertainment industry – film-makers in particular – is preparing for a breakthrough. When making a VR content, intuition is the only guide since the rulebook of interactive storytelling is yet to be written.

Interactive storytelling Jacek Naglowski credits Trend Nomad
Jacek Nagłowski – director, scriptwriter, film producer. Professionally involved in the world of mobile applications since three years ago. Co-founder of Circus Digitalis from Łódź, publisher of digital, interactive, multimedia-enhanced products for tablets. His interests focus on virtual reality and interactive storytelling. Photo: Trend Nomad

Let us first distinguish between 360-degree or spherical pictures and films and virtual reality. These terms are often used interchangeably.
Jacek Nagłowski: In media, and even at film festivals, these terms are commonly lumped together like synonyms, while they actually refer to distinct formats. 360-degrees – or panoramic – photographs and films are made with a single camera rotating on a single axis. The camera records images of a single surface, forming a circular image. By using a cursor on a computer or shifting a smartphone right or left, the viewer changes the angle. Spherical pictures and films, on the other hand, register images in two planes. In a 360-by-360 recording, the viewer can look not only to the sides, but also up or down. What is being recorded is the entire sphere around the camera. These are the images we see in Google Street View. In both cases, we are dealing with a recording of reality produced with a material camera. Meanwhile, virtual reality (or VR) does not require a traditional camera with a lens, but is generated by a computer from start to finish. Not only does the viewer enjoy full control over the direction of viewing, but he can also change his point of view onto the given space in all directions, x, y, and z, at will. Incidentally, VR is not a computer-generated reality – if that were the case, the assumption would be that it is there even when we’re not looking – but a computer-generated image of a three-dimensional reality. What we see in the virtual reality goggles is rendered when we look at it.

In 2016, Oculus, a Facebook subsidiary, will initiate the sale of its first consumer googles designed for viewing content in virtual reality – primarily games and films. Sony, Samsung, HTC, and many other companies are introducing similar devices.

In layman’s terms, anything one sees through goggles such as Rift by Oculus – including 360-degree or spherical pictures and films – is virtual reality.
Yes – the term ‘virtual reality’ is applied to spherical images of the actual world registered with a material camera for the sole reason that you can view them using VR goggles. For me, that’s absurd. Spherical films were made before virtual reality goggles went mainstream without ever being called VR. As publications about Oculus came up in droves, all these terms were thrown into the box labelled ‘virtual reality’. If it were up to me, I would have distinguished ‘immersive films’, that is, stereoscopic spherical films recorded with a material spherical camera. However, I’m afraid this terminological mess can no longer be put in order. The same happened with stereoscopic film and three-dimensional graphics – though neither has anything to do with the other, they are both described as ‘3D’. And that’s how it is.

From the perspective of the viewer, spherical and virtual reality films have at least one thing in common, namely a high degree of interactivity previously known only in games.
Yes and no. Interactivity can take place on several levels. In spherical films and VR animations, interactivity occurs at the level of the camera: the viewer can choose the direction of sight, taking control over the camera. Experiments with interactivity can also occur within the script: the viewer may choose a plot line to follow. This requires that the script is developed in the form of a tree with various paths branching out. Third, interactivity may occur in editing. The viewer may choose when to make the cuts, which camera to choose, or which storyline to follow. Fourth, interactivity can consist in interventions in the represented world – modifications of the appearance of objects or spaces, of relations between objects or characters. The interactivity of the camera is the only common denominator of spherical and VR productions. The other three may occur, but don’t have to.

We don’t know the language of interactive storytelling well enough, we’re unable to use it yet. This is a major field for research.

What happens to the experience of storytelling when the viewer can try his hand on choosing angles or making cuts, activities so far restricted to film-makers?
The Łódź Film School is currently forming a Laboratory of Image Perception. We intend to use it as a platform for studying this problem in hope of finding answers to questions like the one you posed. We’ve only just began. Intuition tells us that an increasingly active viewer is less capable of experiencing the story. A spherical or VR film is like a game: you are either the active party, or a passive witness to events. We believe that reflectivity and the ability to experience the story increase only in the latter case.

Virtual reality allows us to see the represented world through the eyes of a protagonist of our choosing. Does that further empathy?
That’s a very important question. The initial assumption is that entering a protagonist’s body and seeing through his eyes does not imply deeper empathy. Paradoxically, this device actually limits empathy. In my view, that’s because the viewer never identifies fully with the protagonist of the story, and what we’re seeing in only the projection of the feelings of a reader of a book or a viewer of a film onto a character. And that happens only when a part of the story is left untold. Another figure must be involved here. To experience another person’s story, I have to remain myself. When you read a book written in the first-person perspective, you do not feel like you’re the protagonist of the story. Quite the contrary, you feel as if the narrator was sitting in front of you reading aloud a book about his own experiences. The viewer has to be allowed to withdraw and reflect. When you see the world through the eyes of the protagonist, there’s no room for distance or shared experience.

When you dive into a space viewed through virtual reality goggles without entering the body of a protagonist, you feel a greater empathy towards the characters you see than you would when seeing the same scene in cinema, on a TV screen, or a mobile device.

Virtual reality goggles cut the user off from the outside world. Viewing films becomes a personal experience – no one sees the same screen I am looking at.
That was exactly what happened to music when sales of portable cassette players began in 1979. Today, when you want to immerse yourself in the music, even when you’re alone at your home, you will use headphones. Insulation from the outside world intensifies the experience. On the other hand, the introduction of the Walkman also led to another change: music became an object of everyday use, a background to other activities – a soundtrack to living.

Will Oculus become the new Walkman? Will people on public transportation take to wearing VR goggles?
I don’t think we would meet VR-users on a bus, tram, or in the metro. One reason is that wearing an Oculus would make them miss their stop. Besides, I don’t think it would become socially acceptable. Still, such devices will surely be seen on planes or in hotels.

In the United States, virtual reality-based entertainment centres are being raised under The Void franchise. Others will pop up in different locations across the world.

Viewing films with VR goggles is a very intimate experience.
From its inception, cinema sought to isolate the viewer from outside stimuli interfering with the viewing of the film. Virtual reality takes this tendency to the extreme. Here, we are truly alone. If there’s anything that might bankrupt the cinema industry – companies that own cinema halls, not film-making as such – it would be precisely virtual reality. VR can emulate the cinema experience better than cinema itself – even two-dimensional films with framing offer a more immersive experience when viewed on a screen inside a virtual cinema generated by VR goggles than in a real theatre. And no one is munching popcorn.

https://www.youtube.com/watch?v=cML814JD09g

Viewing a film in a cinema is largely a group experience – we are continuously reminded that someone is sitting right beside us, and there’s always someone we can discuss the film with after it’s over. Can viewing films in VR become a social experience?
For the past two years, Linden Lab – the company that launched Second Life – has been developing a VR platform based on social networking, called Project Sansar. The users will share virtual space with other members, or rather, their avatars. Meanwhile, AltspaceVR has already launched its social network. You will meet other figures in a virtual spaced viewed with VR goggles. Each computer character represents a living human being. You would watch games or films together. The images in the virtual reality goggles of all users are synchronised. Besides, virtual reality goggles are fitted with players designed for standard 2D films. They are screened in a virtual theatre. When the viewer looks around the room, he will see seats reflecting the light emitted by the virtual screen. Still, those seats will be empty, which I consider to be a serious drawback. I believe that when you’re conscious of other people in the virtual theatre, whatever avatar is involved, your experience is far deeper than anything known devices for long-distance communication can offer.

Shouldn’t avatars look like people?
For basic non-verbal communication, it’s quite enough if they are human-like. What matters is calibrated eye contact. When you establish that, you will be certain there is actually someone next to you. Communication via Skype seems artificial because we cannot capture the gaze of our interlocutor. That’s because the cameras are located on top of the screens, and interlocutors look at the screens. Virtual reality eliminates this problem because the users look at avatars, not images from cameras.

https://www.youtube.com/watch?v=c1j3WQeS_Qc

Are spherical and VR films suitable for editing?
At first glance, it might seem that there is no room for cuts in films designed for virtual reality goggles, but film-makers from New Deal Studios, the production company currently most advanced in using VR, are experimenting with cuts and claim that editing is possible. A lot depends on what conventions are accepted by creators and consumers. On the one hand, we are facing questions concerning the perception of the viewer and the potential risk that he might be lost when cuts are applied in VR. These issues are being assessed, and we expect to come up with several theories. On the other hand, conventions are a fiercely powerful tool capable of forcing the mind to read certain gestures in a particular way. Traditional films also come up with these questions: the conventions of storytelling, including editing, are different in Hollywood than they are in Bollywood. We no longer tell stories the way they were told a hundred years ago. Whether sharp cuts are acceptable in spherical and VR productions depends on the conventions applied. It’s a process that will take years as it takes many productions and various experiments.

Telling stories in VR requires astounding directing skills, perhaps even greater than in traditional film. One has to know how to command the viewer’s attention, how to make him look in the right direction at the right time.

The film-maker may lose control over the direction of the viewer’s gaze. Is there anything he can do to at least try to direct his attention to areas where something significant is about to happen?
Skilful set-designing is the basic tool, but this can be achieved in other ways, for instance, specific sound effects. When the viewer is looking in a particular direction, he will hear pre-determined sounds clearly, while others fade. When he turns his body or head in a different direction, the sounds he hears will also change in intensity – as happens in real life. But when sound is used for narration, we can point the viewer’s attention in the right direction. We can manipulate him. This is what sound creation is for – the secret of any medium boils down to the ability to accentuate and hide information, which allows us to communicate a story, and not chaos.

Are any special technologies used for recording that kind of sound?
Yes – for instance, VR films are recorded in binaural audio. Special microphones are placed in prosthetic human ears at distances and in mutual relations identical to those of their natural equivalents. The shape of the prosthesis is very similar to that of an actual auricle. The shape of the earlobe seriously affects the way in which the sounds we hear reach us. Binaural sound is listened to on headphones. Since we have learnt to emulate binaural recording in the process of sound creation, we have complete control over the type, direction, and intensity of the sound heard by the viewer.

https://www.youtube.com/watch?v=u163wC6mP2A

Virtual reality isn’t just about feature films. It also includes virtual art or design exhibitions, among others. Is there any point in making faithful replicas of material exhibitions in the VR standard?
In my view, there’s little point in copying elements of reality, including museum halls or exhibition rooms in their entirety, into the virtual world. My views on sound are similar in that regard. The use of any new medium – photography, film, or computer graphics – at its inception always involves the same error: the faithful reproduction of reality. But the media themselves can never reproduce reality – they only transmit images of it. The more conscious the use of a given medium is – like when framing or depth are used to do away with irrelevant information – the more successful the transmission will become. Still, I’m certain that many museums and galleries will welcome spherical cameras and offer digital access to their interiors, including the exhibited works, via the Internet. But the audiences will soon find that insufficient. We have to come up with new ways of narrating virtual exhibitions. They have to be devised from scratch to avoid the banality of a mere survey of exhibits with multimedia attachments or a school trip across three-dimensional visualisations of the premises of a given cultural institution.

We must do away with the notion that virtual reality requires walls, floors, or ceilings. They are not necessary. The only thing that matters is that the space we generate aids the reception of the exhibits in the best possible way.

Where should we begin when designing virtual exhibitions for VR goggles?
We should begin with the exhibition space. The curator is unconstrained by any limitations of a determinate location. The hypothetical, quasi-material space can be as big as you want – it can be infinite. On top of that, virtual reality permits an immediate change of location – changing of the point from which a given object is seen as well as movement between subsequent exhibition spaces. We need to learn how to direct viewers across a virtual space from one object to another. The greatest and most trying task of the curator of an exhibition in virtual reality will consist in defining the exhibition space from start to finish – not in filling out the cubic metres he is assigned to.

What sources can the curators turn to if they want to learn how to use virtual reality?
Experimenting is the only way – trying what works and what does not. The curator takes absolute control over the space that is being created. He can do anything he wants, so long as it helps him tell the story. It’s a freedom curators never enjoyed before – exhibitions were placed in enclosed spaces, with the objects placed on the floor or mounted. In VR, these elements are gone. One exhibit can penetrate through another. I find this randomness scary. It’s not freedom, but chaos. Freedom is when you have a choice. One has to be very careful to prevent excessive information from adversely affecting the response to the exhibition.

Preparing a film or exhibition in virtual reality is deceptively similar to work on a theatre play, the only difference being that there are no restrictions resulting from the dimensions of the space in use.

How should we talk about a material object that can be viewed from any perspective, favourable or not, in virtual reality?
We are about to conduct research into the human perception of objects, into what draws the people’s attention in the first place, using eye-tracking. The first trial will involve studying persons not professionally involved in viewing objects. Then, the same experiment will be conducted on a group of designers – we are interested in how they look at objects they don’t know. Third, we will analyse the reactions of photographers and camera operators. When we compare these lines of the gaze, we will know how people apprehend information about objects, whether there is any persistent rule concerning object perception and which line is the most common. We will learn to tell stories about objects and draw paths around them in virtual reality. The viewer will feel free to choose his point of view even as he is led toward the key lines of the gaze. It’s not about favourable or unfavourable appearances – that’s for marketing to tackle. The main task is to distinguish the ways of looking at an object that convey the idea of it in the most fruitful and natural manner, leading to understanding.

How can we eliminate irrelevant information?
Outside of sound designing, we can use depth of focus. Before the première of Tron, people believed that everything had to be razor sharp in 3D. When they saw the film, many film-makers changed their minds. The unfocused bits of the screen work perfectly well, but this is not the regular lens diffusion. We have learned the use of a new tool. The same will undoubtedly happen with films in virtual reality.

Do spherical cameras allow for changes of focus?
No, this isn’t technically possible during filming. Diffusion can be added in post-production through computer editing on a frame-by-frame basis, a partly automated process.

How does it work with images generated electronically in entirety?
Focus can still be manipulated, of course, but no one knows how to employ consciously the effect in VR films yet.

Can information media become fully immersive, too?
Though telling stories about objects in virtual reality is not particularly easy, VR is perfectly suited to providing reports from the field. Experiments in immersive journalism are being conducted today. One example is the VR-based Project Syria – Nonny de la Peña transports the viewer into the cruel world of war. The viewer first witnesses the bombing of the central area of a Syrian town, and then moves to a refugee camp. This three-dimensional world is computer-generated, but the visualisations are based on films, pictures, and sounds recorded in real life. In 2014, the film received a standing ovation at the Sundance festival. Documentary materials recorded with a spherical camera will make an even greater impact, particularly if the image is transmitted live. Of course, you don’t have to use this technology just for disasters. It can just as well be employed for transmitting material of much greater levity, such as sports events. NextVR, an American company, is already on the case.

https://www.youtube.com/watch?v=jN_nbHnHDi4

Will design magazines, including those about interior design, have their own channels at VR websites?
The readers of interior design magazines may feel inclined to view the interiors of apartments in a spherical format instead of just looking at a bunch of static pictures. It might seem to them that this would allow them to catch many more appealing sights. The truth is that a spherical camera would expose the amount of trickery involved in magazine photos. Perfect framing, lighting, some Photoshopping, and the interior becomes picture-perfect. A spherical photograph or film would take away the charm. If I were a publisher, I’d keep away from that format.

And yet some hotels have already published spherical pictures of the rooms on offer on their websites.
This is done for a purely utilitarian purpose. 360-by-360 recordings perfectly capture the appearance of the rooms, which is going to be verified by the guests upon arrival. You won’t see that transparency from magazines about interior design. They are not meant to provide images of the real world, but to beautify it.

An apartment that looks beautiful and spacious on a well-framed picture might turn out to be a cubby hole in virtual reality goggles.

Is there no Photoshopping in spherical imaging?
You may clean up a spherical picture, obviously, but you can’t trick the sense of scale or proportion in VR. Pictures in magazines, like in any other media, always present objects or spaces in a specific scale, usually reduced. A few tricks is all it takes to manipulate proportions. What’s striking about virtual reality is the sense of actual scale. That’s something no older medium could give you. In a 1-to-1 three-dimensional picture, proportions and distances are so real that the viewer almost senses the physicality of the objects. Computer-generated objects don’t even have to look super-real for the viewer to feel their materiality. In the case of images of living beings, the sensation is even stronger. The sense of materiality, particularly when eye contact is established, causes us to empathise with them. Chris Milk is an expert in generating empathy with virtual reality. You should see his TED2015 talk.

https://www.youtube.com/watch?v=iXHil1TPxvA

Today’s film scripts contain certain pre-defined elements: the first and second turning point and the denouement. Do the same rules apply to VR films?
Film-makers know all about using writing, editing, framing, and sound to evoke emotions in viewers of traditional films. Things are far less clear when it comes to telling interactive stories. All art, and particularly narrative art, affects people only to the extent that its creators can exert control over the emotions of their audience. In the case of VR documentaries such as Immersive Journalism that can be achieved quite easily. Feature films constitute a greater challenge in that regard. One has to experiment: only when we become aware of the medium will we create works of significant cultural value, the kind that move us and speak to us of important matters.

“Wired” named its most important film of 2015, and it’s “Henry”, an animated VR film. It’s a short tale of a hedgehog that likes to hug but can’t because of its spikes. He looks at the viewer frequently, sparking feelings of sadness or excitement.

Does the viewer of a film seen through goggles realise the existence of the screen and a material or imagined camera between him and the characters in the film?
The viewer knows about the camera so long as he sees a screen, or rather, its boundaries. In virtual reality, the user is armed with goggles and does not see any screen, and thus forgets about the camera. He is thrust onto the set.

https://vimeo.com/134754691

In the 20th century, TV sets made an impact on house furnishing – the receiver was the centrepiece of the room. The viewers sat almost motionless on a sofa or a recliner, gaping at the screen. Where and how will we view VR films?
The position of the body deeply affects our response to content prepared in virtual reality. The agreement between body movements and what is represented inside the goggles may decide whether we are engaged in the represented story. At the same time, the user has to feel safe and comfortable in spite of being shut out from the outside world. While VR changes as a matter of course, the furniture we use when watching content in virtual reality should be stable. I can’t imagine the actual appearance of such a piece of furniture, but I would like it if a Polish furnishing company entered the market with the world’s first collection of furniture designed with VR users in mind.

In 2016, we will see whether virtual reality is a fad or a business that can change the entertainment industry for years.
VR might share the fate of rollercoasters. It’s great fun, but best used from time to time – it is too physically and psychologically discomforting. Still, I believe in that scenario. This cinema is only just starting to develop.

 

If you want to ask Jacek Nagłowski any additional questions concerning the production of virtual reality films, send him an e-mail at jacek.naglowski@circusdigitalis.com.

www.circusdigitalis.com

This interview was conducted in September 2015 as part of a final thesis at the post-graduate programme “History and critique of design” at the SWPS University in Warsaw under the direction of Agata Szydłowska and Monika Rosińska.

Do you like the interview? Then buy me a coffee! You can donate a small sum of money using your PayPal account or credit card. All donations will finance my journeys to fairs, festivals and conferences devoted to design and new technology – this is where I find news for my blog. Just click the button below to perform a secure transaction. Thank you for your support, it will help me to take a step forward and write new posts.

More articles