Essential Evaluators: Snapshot | Data Decisions

Posted on Dec 16, 2020

A graphic reading "Essential Evaluators" with the subtitle "Snapshot: Data Decisions" — Collecting data is one thing, but figuring out how to apply it can be even more of a challenge. Here's how museums across the country are putting research into practice while adjusting to COVID-era operations.

Essential Evaluators seeks to gather evaluators in a common space to dialogue, reflect, and support each other in a world upended by COVID-19 and the Black Lives Matter protest movement. This is a time of uncertain and unknown expectations in our professions, in our institutions, and in our communities. We invite you to join us as we rethink, revision, and ultimately redefine our roles as evaluators and our place in museums.

Other posts in this series:

Our first “Snapshot” post was published on September 16, 2020. That feels like a lifetime ago at this point. Since then, we’ve lived through a historic presidential election, another COVID surge, and a new round of reporting on museum closures and staff impacts. In the middle of these monumental cultural, political, and economic changes, we thought it was a good time to check back in with the field and bring you a second Snapshot.

In the first edition, Snapshot: Data Collection, we focused on the innovative ways museum evaluators were collecting data during a time of institutional closures, social distancing, and new safety standards in response to COVID. However, as one commenter observed, “We know that about 95% of US businesses collect feedback yet only 10% act on it—so hoping these ideas will include an action module to assure future visitors a better experience.” We couldn’t agree more, and so we reached out again to the field to bring you the second in our Snapshot series, Data Decisions.

For this Snapshot, we asked museum practitioners to share stories of how their institutions have used data collected during COVID to assess their work, make changes, and think about programs and policy. The stories that came in fell into three categories: institutions used data to 1) assess visitor comfort and experience with new COVID precautions, 2) gauge visitor interaction with social media, and 3) inform frontline staff training. We want to thank the Shedd Aquarium; the Museum of Science, Boston; the Cleveland Museum of Art; and the Minnesota Historical Society for sharing their stories with us.

Visitor Comfort and Experience

Amid the sudden changes to visitor experience, some institutions have used data to make visiting as safe, comfortable, and enriching as possible.

Like many organizations preparing for reopening, the Shedd Aquarium revised its pre-COVID exit survey, adding twenty new items to help understand how guests prepared for their visits, how they felt about safety modifications, and how effective the Shedd was in implementing necessary safety precautions, such as providing hand sanitizer, cleaning public spaces, and limiting capacity. Lindsay Maldonando, Director of Research and Evaluation, was able to accommodate the new survey items by eliminating questions that were no longer relevant to the new guest experience, like those about programming that needed to be canceled for physical distancing requirements.

Making data more accessible to frontline teams was a significant priority for Maldonando. She provided the Guest Relations team access to a SurveyMonkey data dashboard, so that they were able to easily review data daily and adjust operations where necessary. Guest Relations leadership also received a link to view survey responses, which included aggregated results for all questions, including frequencies, mean ratings, and open-ended comments. In conjunction with other data analysis tools like Tatvam (a platform that analyzes online reviews from Facebook, Trip Advisor, Google, etc.), the Guest Relations team was able to make quick data-informed decisions to improve the guest experience. For example, when they saw that guests rated their ability to physically distance at the aquarium lower than the Shedd’s benchmark, they quickly pivoted operations to include more focused attention on promoting distancing.

In addition to providing insights about Shedd’s preparedness, this data offered other important operational insights. For example, guests’ self-reported stay times allowed the Shedd to amend the number of timed tickets sold per hour, which helped increase revenue against original projections. Data also demonstrated that some concerns raised in planning to reopen, e.g., timed tickets, theater safety, and mask-wearing, were not actual concerns at all. Additionally, data could be shared with key stakeholders both internally and externally, including government officials and other organizations preparing for their own reopening, to provide a concrete picture of preparedness.

Much like their colleagues at the Shedd, staff at the Museum of Science had a lot of questions about what visits would look like, how visitors would behave, and how visitors would feel about their visits when they were welcomed back to the museum in July. To help answer those questions, Elizabeth Kunz Kollmann, Manager, Research and Evaluation, and her team focused on collecting data in two ways: conducting extensive observations of the museum’s exhibit halls, which we will focus on here, and collecting exit surveys from visitors.

The Museum of Science has historically used audience observation to study how visitors move through exhibitions or use specific components, to understand what is engaging and where usability issues arise. As the museum reopened, though, Kollman used observation to answer a different set of questions:

Are visitors wearing masks as required? If not, who isn’t wearing masks?
Are people using hand sanitizer? Are our hand sanitizing stations full?
What exhibitions are people visiting? Are they using exhibits differently than they did prior to the pandemic?

To answer these questions, staff from across the museum, both evaluators and non-evaluators, conducted sweeps of the exhibit spaces every day for the first few weeks of reopening. They made sure to observe each exhibition every day and at different times of the day. From this, they learned that most visitors were wearing masks, though their younger visitors were having the hardest time with this new rule, and that very few visitors used hand sanitizer. These observations led to more frequent museum-wide announcements reminding visitors to wear their masks and clean their hands throughout the visit. They also learned that the public was heavily using two exhibitions while others were much less visited, which allowed for focused cleaning efforts in the most popular exhibits and helped to alleviate some of the extra work expected of exhibit maintenance staff. Finally, they found that visitors were still touching interactive exhibits and using them as they did before the pandemic, which made Kollmann confident that the museum had made the right decision in keeping the interactive exhibits on the floor.

At Minnesota Historical Society, Hayes Scriven, Site Manager, and his team had to decide whether to reopen Split Rock Lighthouse, a National Historic Landmark that annually sees 160 thousand visitors a year. After reviewing safety guidelines, the team decided that indoor experiences, such as the movie theater and gallery, would remain closed, but that an experience could be crafted for the outdoor portion of the site.

Previous visitor data indicated that many visitors preferred to explore the site on their own, so Scriven moved from guided to self-guided tours, created additional outdoor signage that provided historical context, and placed roaming interpreters in key areas to answer questions and give presentations. This combination of interpretive signage and roaming interpreters meant everyone spent less time in large groups, and it allowed visitors to engage with staff on their own terms while enjoying the site at their own pace. Though many visitors have asked why they couldn’t go into the tower, survey data shows that they have been finding their visits as enjoyable as in years past. In addition, staff say the modified experience has reduced stress and added flexibility.

Visitor Engagement with Social Media

Digital engagement has become crucial during the pandemic, and data can help with the difficult task of finding the right approach.

When the Cleveland Museum of Art began its temporary closure in mid-March, its primary audience suddenly became its online audience—which it had long sought to better understand but never found ample time to dedicate to ongoing evaluation. The museum’s closure gave Hannah Ridenour LaFrance, Research Manager, a unique opportunity to step back, assess available metrics and data sets, and chart a path forward for ongoing, systematic analysis of the museum’s social media reach that could be continued even after it reopened. At the same time, the Communications & External Relations team was experimenting with a variety of social media strategies to engage and grow the museum’s online audience.

The research team set up the first phase of an exploratory study to determine what data points could potentially answer these questions:

What audiences are we currently meeting/reaching on social media?
How are we currently engaging our audiences on social media?
What audiences do we want to be meeting/reaching on social media?

Across April and May, they reviewed the “top” posts every week on the museum’s Instagram, Facebook, and Twitter profiles, noting the number of likes as well as keywords used. While services like Sprout look at higher-level indicators such as impressions, LaFrance wanted to allow for a qualitative analysis through examining the content as well. The data supported expectations about the success of commemorative days such as artist birthdays, as well as common hashtags used during the pandemic such as #MuseumFromHome. They also revealed which social media strategies were less successful in garnering engagement.

With so many data points to consider, phase one of the study enabled the Cleveland Museum of Art to determine which were most telling, as well as to establish how to record certain metrics for adequate and ongoing analysis and insights. But, as with any good evaluation, it also identified further questions to answer. Now in phase two of the study, LaFrance is switching from inductive to deductive coding, reviewing social media follower counts for each platform monthly, as well as recording top weekly posts through their likes/favorites, content origin, keywords, and category. To record impressions and other high-level metrics, the team is still using Sprout.

LaFrance has also turned to social media for data collection for some projects, such as a title-testing instrument for an upcoming exhibition. The data from that study, while not a completely representative sample, gave the team a sense of their audiences’ visitation habits and membership status on specific social media platforms. The exploration into social media metrics has been a good example of how disparate data sets can still work collectively to create a stronger impression of a museum’s outreach and communications efforts, and has also served as a blueprint for how metrics tracking can evolve alongside the museum’s ongoing and valuable social media strategies and needs.

Frontline Staff Training

Frontline staff have experienced the most changes to their jobs this year of anyone, and data can help ease them into the new procedures.

When the Minnesota Historical Society began its phased reopening with two of its sites, David Sandager, Program Specialist for the Interpretive Programs Work Unit, knew it was important to develop a training program to prepare frontline sales and interpretation staff. So he created Return to Open, a training program focused on institutional updates, site-specific COVID safety measures, and tips for working with the public during a pandemic.

Looking ahead to all fourteen sites reopening, Sandager wanted to hear from the first staff members who went through this new training program, so he administered a post-training survey. The survey asked staff to reflect on what in the training had actually been helpful now that they had been interacting with the public for several weeks. It asked three open-ended questions (example responses are included):

What worked well for you about this training?
- Online components that we could read and complete at our own pace, and followed up with in-person Q/A of site particulars
- Hearing that staff safety is a high priority
- Laying the groundwork for the new COVID situation
What would you change about this training?
- Provide some information about the lessons we are learning at the site that lead to success. What have we learned from guests how to keep them safe and yet do our very best to make the experience a memorable one enhancing their understanding of the history of the site and enjoyment of the opportunity
What advice would you give your colleagues at other sites regarding returning to working with the public?
- Be friendly but firm in letting the public know it is for the safety of the site and their safety, so we can be open for them.
- Be prepared for plans to be tweaked on a regular basis and for the public to not follow/read signs. 🙂
- Don’t let the small number [of] disgruntled visitors get you down. Most visitors will be very understanding about changes at your site due to the pandemic.

The data revealed structural elements that needed to be remedied and identified new information that would be crucial for future trainings. Subsequent trainings highlighted the advice elicited from staff for their peers at the next sites to open, who reported a great degree of appreciation for tips for working with the public. Hearing that their colleagues could make it work and felt supported helped allay some of their trepidations about returning.

For future Return to Open trainings, Sandager and his team are addressing concerns raised about anonymity, by eliminating questions that asks about department and position, and are grappling with whether advice offered by staff at an outdoor sites will be applicable to staff at indoor experiences. As the team thinks about how best to integrate staff advice captured from the surveys, they are also exploring additional survey questions to add.

These are just a few examples of how evaluators are working in this COVID world, adjusting their programs based on data, and seeking to best serve their visitors. We know there are hundreds of other examples out there. Share your experience with making data-based decisions during COVID in the comment section to keep the discussion, ideas, and learning moving forward.

Note: As of December 2020, Rose Jones, PhD, has resigned as a co-curator of Essential Evaluators. We thank her for her contributions to this effort. Laureen and Andréa look forward to continuing to give voice to the evaluation community.

Laureen Trainer and Andréa Giron Mathern

AAM Members get exclusive access to premium digital content including: