When Rob Stein, AAM’s Executive Vice President & Chief Program Officer, first approached the AAM Media and Technology Professional Network with the announcement that AAM and the Knight Foundation were planning to hold a series of convenings on technology topics, the first to cover augmented and virtual reality, we were delighted to help out.

Diving into a topic that has confounded the expectations of so many seemed a bit daunting, but nobody has a broader sense of what issues are most important to the field than AAM. And the Knight Foundation has not only underwritten interesting museum experiments with digital technologies but has funded staff positions to allow museums to explore these technologies while building institutional capacity and experience. The Detroit Institute of Arts received funding from the Knight Foundation to host the Immersion in Museums: AR, VR or Just Plain R? convening, and made their Lumin app available for attendees to use in the museum!

Getting off the AR/VR hype treadmill

In developing the content for the convening, one of the first conversations we had focused on whether we needed to include reality alongside augmented reality (AR) and virtual reality (VR). AR and VR have been generating a fair amount of interest and associated news stories in the past year. Detroit Institute of Arts’ Lumin app (more on that later), and SFMOMA’s AR experiments in their Magritte exhibition are just two examples. However, we were also conscious of a new crop of immersive museum-like experiences that were just as interesting. For example, MeowWolf in Santa Fe, immersive theatre productions like Sleep No More, and all-digital experiences like the TeamLab in Japan and Atelier des Lumieres in France. The thing that tied all these disparate projects together was our sense that they all provided visitors with a sense of immersion.

The decision to use immersion as the lens to look at examples of physical and digital experiences was important and allowed both participants and presenters to question the technologies differently than if the focus had been solely on the platforms. Moreover, and more importantly, by being about immersion, it constantly pointed us back at visitors’ wants and needs, rather than at what was possible using today’s technologies. I think the success of that framing bodes well for any future convenings AAM holds.

This leads us to the actual content of the convening.

MeowWolf, art and letting people rediscover their inner scientist

On day one, Golda Blaise and Vince Kadlubek, two of Meow Wolf’s founders, started off with a discussion of both what Meow Wolf is, how it has grown and evolved, and how they approach creating immersive art experiences. And, they were provocative, to say the least, in a good way. To watch the keynote presentation, please see: How Immersion Makes Art Accessible.

Hands-on with Lumin, DIA’s AR app

In the afternoon, participants heard from DIA staff who developed the museum’s mobile tour app, Lumin. Using Google’s Tango technology and marrying 3-D mapping of the building with augmented reality, the Lumin app steers visitors through the galleries and provides new ways to engage deeply with objects in the DIA’s collection. As part of the workshop, the Lumin team presented their experience building and distributing a smartphone based wayfinding/content app. Demonstrating a new technology in front of your peers is never easy, and DIA staff deserve great credit for their bravery and willingness to share their experience, warts and all. It was a great example of institutional commitment to advance themselves and the field.

Afterward, we broke into groups and actually took Lumin for a tour. So often, talk about digital technologies takes place without the actual technology present, so being able to see what it was like to be a visitor to DIA navigating with Lumin and exploring its features was incredibly helpful. In groups of two or three, we spread out across the museum, following the app’s suggestions for galleries to visit, objects to explore, and trying, mostly successfully, to follow the blue line on the screen that told us where to go.

Determining indoor location and wayfinding on a mobile device are thorny issues that have consumed developers for years. The promise of delivering the outdoor, GPS-based location – the ubiquitous blue “You Are Here” dot on phone maps – has been a difficult experience to deliver until now. And Lumin delivers a pretty smooth wayfinding process. Our group was able to move around the museum, choose places we wanted to go and get there without any more fuss than if we were using a phone to navigate outside. Which is to say it worked, most of the time. Sometimes we’d get hung up in doorways, turning in circles trying to find our path again, and sometimes we’d just lose the signal completely.

Skip over related stories to continue reading article

Where I thought Lumin really shone was in the galleries as a tool to uncover different kinds of content about the objects chosen for augmentation. At an African ceremonial urn, the app showed the carving process, starting with an AR stump that floated to one side of the object. The carving process is recreated, turning the stump into an urn and revealing details that you might miss, like a wooden head that is carved inside the vessel and could roll around. That hint of motion makes it easy to see and makes the stationary head in the real object take on new meaning. At a Seurat painting, you could explore his pointillism by digitally zooming in on the canvas without getting too close to the actual painting, or incurring the wrath of the guard. Looking at a mummy and being able to digitally peer inside the wrappings to see the skeleton inside was really compelling. It was a great example of one of the points that Andrea Montiel de Shuman, the Lumin Project Manager, made in her presentation, namely that the technology doesn’t have to be perfect to be meaningful. The mummy is currently the favorite stop for the public and due to light level restrictions is also where the phone has the most trouble finding its location. These glitches don’t seem to bother visitors, though, because looking inside the mummy is so meaningful to them.

For more on Lumin: https://hyperallergic.com/350320/more-screens-more-knowledge-testing-the-detroit-institute-of-arts-new-augmented-reality-app/

‪‪The Daily 360 VR, or finding the balance between scrappy and ambitious

Day Two began with another dynamic presentation by Maureen Towey, a polymath who has developed immersive theater, done interactive video for the band Arcade Fire, and who was the producer behind the New York Times’ “The Daily 360” VR feature, where the Times published a piece of 360 video content, daily, for fourteen months. As someone who produces video for a living, it’s hard to overstate how remarkable an accomplishment that is, and what kind of courage and sheer determination it would take to accept that job and do it so well. Like Golda and Vince’s talk, it was full of insights and copious examples to illustrate her points.

Maureen Towey leading us through her work of VR and Immersion – @NYTVR , @arcadefire, and immersive theatre #ImmersionInMuseums @AAMers pic.twitter.com/8YWXdrFKXO

— Rob Stein (@rjstein) September 7, 2018

To watch the keynote presentation, please see: Immersive Storytelling at The New York Times.

What issues arose?

One of the highlights of the event for me, in addition to the speakers, was the opportunity to really discuss the topics with small groups of colleagues at a level that’s hard to achieve in a typical conference setting. This model of convening 75-100 people for a couple of days to look at a tightly defined topic holds great potential to build capacity in the field. The breakout sessions were all hosted by AAM and M&T representatives who served as notetakers and guides. Going through the notes, a few themes arose that spanned the whole group.

Cost

The cost of tackling new digital initiatives was a pain point for many in the workshop. The widespread view was that non-digital experiences are still cheaper to build, though the contrarian viewpoint was raised that physical experiences are just as expensive, just differently expensive since we are used to spending money to build things, buy casework, paint, and print catalogs.

Attraction vs. Distraction

The issue of new technologies being a distraction was a common worry. AR/VR has the potential to distract from the rest of the visit. Some participants felt that the “wow” factor, the novelty of doing something new, had the potential to drive the experience, and not the content. The form factors of AR and VR platforms, headgear vs handheld, also come with usability challenges. Headgear is distracting and can make people feel self-conscious. Handhelds restrict the user to a narrow field of view but benefit from the fact that most people are used to holding a phone. DIA’s use of a pistol grip phone holder was an interesting response to a challenge with the phone form factor; your hand gets tired-fast.

One participant wondered whether “retro” museums with no digital enhancements might become cool. There is a small but growing number of museums that proudly ban photography. Might others follow suit as a way to let visitors “disconnect”? Another predicted that black box VR spaces will appear akin to Star Treks’ holodeck, where you can step in and swipe through the universe. A third imagines both would come to pass and coexist.

Agency

This word came up a lot, especially in relation to how proscriptive most digital experiences are. People wondered whether AR and VR experiences could create the same kind of dynamic interplay between visitors’ conceptions of self and to the objects they’re encountering in the real world. There was also conversation about the merits of wayfinding and planned experiences versus more “free range”, visitor-driven explorations.

On-boarding and Off-boarding

This made me think of something that came up in all three talks; an emphasis on onboarding and off-boarding that is largely absent in museums. In MeowWolf, the house is a training ground for setting visitor expectations about how you can behave in the space and what can happen. The Lumin app experience starts with a human showing you how to use it, and gauging what kind of help you might need. The Daily 360 made all their videos available to users sitting at a computer, as a way for them to experience something similar to the VR experience. This tutorial stage, like those in a video game, is an important tool.

I’ve long wanted to make a “How to visit the museum” gallery for the museums I’ve worked at. The experience in the galleries may be the peak or climax of the visit, but what comes before it and after are just as important.  The New York City venue for Sleep No More gives playgoers a tutorial beforehand and encourages them to visit the bar afterward. This is not just to get them to spend money on drinks, but so visitors have a chance to decompress, process, and share stories with each other of what they all experienced. That way, the divergent, fragmentary personal experiences are combined into a larger understanding of what Sleep No More is about in a way that doesn’t detract from the experience while they’re in the thick of it.  It may also have something to do with their success in encouraging repeat visitation in a city full of new things to do. What might your museums’ onboarding and off-boarding experiences look like?

Where is the Big Reveal?

Narratives often have a “big reveal”, a moment where an important new element of the plot is revealed to the reader or viewer. MeowWolf has a big reveal. Is there ever a “big reveal” in a museum?

Worldbuilding

One thing MW, SNM and NYT 360 all have in common is that they are creating complete worlds for users to experience, not engaging in space planning exercises with a list of objects. Vince posited that this a major difference between what they do and what museums do. Our work always starts with fragments and scattered pieces, from which we attempt to construct a narrative. Objects are arranged in a space and supplemented with interpretive materials to create a space. MeowWolf, immersive theatre, and VR experiences create the entire world the user experiences, and as such, may have a much easier time of creating that illusion of place.

What is “Immersion”?

Given the title, it’s not surprising that people talked a lot about what immersion is or isn’t. What is noteworthy is that no definition of the term arose from the conversations. What we mean when we talk about immersion is a bit of a moving target, and Seema Rao did a great recap of her experience of the workshop which is worth checking out. Her dictionary defined immersion as “deep mental involvement in something.”

Vince Kadlubek thought the term was too loose and preferred to talk about what MeowWolf did as creating alternative realities, and embodied experiences for visitors. This was echoed by many others, this idea of immersion as a fully embodied experience. Too often, museum spaces seemed to be designed for disembodied brains with eyes, who float around the museum and never need to sit down.

For my neuroscientist colleague, Ted Asher, what we were calling “immersion” seemed to correlate to the notion of “presence” – the feeling of “being there”– wherever and whatever “there” is. This sense is created through the interplay of two illusions; the place illusion (a sensation of being in a real place) and the plausibility illusion (the illusion that what your eyes are seeing is really happening).

One group listed their favorite immersive experiences, which included the “Follow the North Star” tour at Conner Prairie Museum, where visitors assume the role of fugitive slaves. An activity at The Immigration Museum, in Melbourne, Australia, where visitors play an early 20th-century immigration judge and have to decide who can and can’t enter the country based on a rigid set of criteria was another. The Museum of Old and New Art in Hobart Australia, where visitors are prompted to share their emotional response to every artwork on display, another. And the Museum of Jurassic Technology in Los Angeles, has a peculiar alternate reality visitors step into when they enter. The examples people referenced intentionally elicit an emotional response from viewers. Could this be part of what makes something immersive? Do museums design interpretation to emotionally impact visitors or deliver intellectual experiences to emotionally detached visitors?

Conclusion

There was a consensus among both the speakers and attendees around the idea that “immersive” experiences involved building mental and emotional engagement with visitors, designing experiences that recognized the embodied, multisensory ways visitors take in the world and gave them an authentic sense of agency to chart their own course through the experience. MeowWolf has demonstrated the appeal of deeply immersive experiences with an incredibly broad audience. DIA’s work with Lumin continues to break new ground in field-testing ways of engaging visitors with AR. The Daily 360, though it was housed in a large institution, used a small team and limited budget to produce a tremendous body of work over a short period of time, and learn a ton about what it takes to make compelling VR content.

Thousands of invaluable art objects at @DIADetroit, but today this doorway has my heart. #immersioninmuseums pic.twitter.com/TfpGui3Ntd

— Chris Barr (@heychrisbarr) September 6, 2018

To learn more about this event, please see: Immersion in Museums: AR, VR or Just Plain R?.

---

Ed Rodley has worked in museums for over twenty years. He is currently Associate Director of Integrated Media at the Peabody Essex Museum in Salem, MA, where he manages a wide range of digital media projects. His recent print publications include “CODE|WORDS: Technology and Theory in the Museum”. He blogs about museum issues at “Thinking about Museums.”