This article originally appeared in the March/April 2022 issue of Museum magazine, a benefit of AAM membership.
A new national study is quantifying the social impact of US museums.
As a museum professional, have you ever wondered how you can demonstrate the impact your museum has on the lives of your visitors? Perhaps, like us, you have intrinsically felt that your museum is making a difference but have lacked the evidence to demonstrate it. These were the thoughts going through our minds when we conducted the social impact studies in Utah that we discussed in our “Brace for Impact” article in the May/June 2019 issue of Museum.
In that article we outlined two social impact studies. The first study was conducted internally at Thanksgiving Point, a multi-museum complex based 20 miles south of Salt Lake City in Lehi, Utah. For the second, Thanksgiving Point and Utah Division of Arts & Museums (UA&M) scaled up the study to measure social impact at eight museums throughout Utah. Both studies found that museum-goers were positively impacted by their visits, meaning their museum experience improved their perspective on lifelong learning, relationships, and personal well-being.
With our Thanksgiving Point and Utah study results in hand, along with an extensive document of “lessons learned,” our team wanted to expand this project and work with museums across the country. So we sought out the Institute of Museum and Library Services (IMLS) and applied for a National Leadership Grant for Museums.
Now, thanks to funding from IMLS, Thanksgiving Point and UA&M are scaling up again, from the Utah study to a national social impact project: Measurement of Museum Social Impact (MOMSI). Working with 38 museums across the United States, we are testing the study methods, further validating the social impact survey, and developing a toolkit so that other museums can measure social impact.
What Our Previous Research Found
For the first study, families were invited to visit Thanksgiving Point for free four different times over a multi-month period. The results showed that families were indeed being impacted positively. The most meaningful aspect of participants’ experiences was spending quality time with their family, and participants were surprised by the quality, size, and variety of experiences Thanksgiving Point offered.
We shared the results with our colleagues throughout the state of Utah, including with our peers at UA&M, which led to the second study, a partnership between UA&M and Thanksgiving Point. Rather than focusing on a single institution, the Utah social impact study included eight museums throughout Utah. As a team and with the participating Utah museums, we rewrote the logic model originally governing the Thanksgiving Point study, expanding the long-term outcome of “strengthened families” to “strengthened relationships,” changing “higher educational attainment” to “continued learning and engagement,” and adding a long-term outcome of “intercultural competence.” We also addressed and reworked some of the social impact survey indicators to align with the new and edited long-term outcomes and developed marketing strategies for recruiting participants and retaining them through the visit and survey process.
The Utah study collected surveys from nearly 400 museum-goers, whose responses demonstrated a statistically significant positive change in 96 percent of the social impact indicators after visiting a museum. Indicators that showed the greatest difference include: “I can see how exploration leads to learning” (continued learning and engagement outcome), “I take time to relax” (health and well-being outcome), “I learn new things from people who are different than me” (intercultural competence outcome), and “I regularly participate in bonding activities with my friends and/or family” (strengthened relationships outcome).
Representing the Field
The process of selecting our 38 host museums for the national social impact study involved multiple spreadsheets, numerous Zoom meetings, a healthy amount of compromise, and maybe a dash of magic. We recruited museums through national professional organizations like AAM, the Association of Zoos and Aquariums, and the American Public Gardens Association; regional museum associations; blogs; and direct outreach. We ended up with a wonderful pool of prospective museum participants.
In selecting museums, we focused on representing as wide a geographic swath of the United States as possible. We are pleased to report that participating host museums span the country, from Norwich, Vermont, to San Diego, California, and from Bellevue, Washington, to Miami, Florida.
Additionally, we wanted our cohort to represent a diversity of budget and staff sizes, allowing us to tell an accurate story about US museum impact. We want the toolkit we will build from this project to be useful to museums large and small, those with an all-volunteer staff and those with research and evaluation teams. We also felt that having participation and feedback from rural museums, or those whose communities are small but engaged, is key to understanding how small museums handle conducting this kind of research and what support is needed.
In selecting host museums, we also considered types of museums. We wanted to represent the broadest possible definition of museum and have included public gardens, zoos, and aquariums in addition to art museums, history museums, and children’s museums. In the end, this consideration may allow us to sort and parse our data along several different dimensions to explore subsets of museums in the US.
Creating the Survey
Our team recognizes the multitude of ways to define social impact, but through each iteration of our research, we worked from the definition that social impact is the effect of an activity on the social fabric of the community and the well-being of the individuals and families who live there. Building on the work of the Thanksgiving Point study and Utah study, MOMSI is measuring social impact through four long-term outcomes: continued learning and engagement, health and well-being, strengthened relationships, and valuing diverse communities.
Through a logic model, we outlined direct connections from the long-term outcomes to short- and intermediate-term outcomes. Short- and intermediate-term outcomes were informed by a rigorous literature review and influence the indicator statements visitors respond to on the social impact survey. (See sample indicators for each outcome at left.)
As mentioned above, Utah museums added the “intercultural competence” long-term outcome, but as we were facilitating host museum training for MOMSI nearly four years later, museum staff felt that terminology was outdated and may not align to what we were really measuring. After discussions with host museums and the Utah Division of Multicultural Affairs, we reworded the outcome to “valuing diverse communities.”
At the conclusion of the Utah study, the social impact survey was found to be reliable and valid. However, it was also very long. We use a retrospective post- then pre-survey design, meaning participants respond twice to each statement in a single survey: once describing how they felt before their visits and once for how they felt after their visits. The Utah study survey included 104 statements. Based on feedback from survey respondents, we knew survey fatigue was a real issue, and we needed to resolve it for the national study. Using psychometric scale analysis, we ultimately reduced the survey to 48 indicator statements.
Where Are We Now?
At the time of this publication, MOMSI host museums are in the midst of welcoming their participating visitors to their museums. Visitors have through May 2022 to complete three visits to their respective museum. After completing their required visits, participants will take the social impact survey, which is administered by our project team. Preliminary results are expected in May 2022, with final results in August 2022.
After their findings are analyzed, each MOMSI host museum will receive a report of its social impact and will have time with our team to discuss how best to use the report and its findings. Through national conferences and publications, we will share aggregate data from all 38 host museums so that we as a museum field can begin to understand our collective impact.
Using data from the national study, the social impact survey again will be tested for reliability and validity. The survey will then become the foundation for a free and accessible social impact toolkit (expected in early 2023) for museums that will include guidelines for participant recruitment, sharing the survey with participants, and how to use the results; resources, including the survey and how to analyze the survey data; and tips for success. To create the toolkit, our team will work with staff from the host museums and reflect on our own lessons learned.
Although MOMSI has closed applications for host museums, we encourage you to follow the project’s progress at museumsocialimpact.org, chat with our team, and watch for presentations and workshops at national and regional conferences. We believe that all museums, big and small, should have the opportunity to measure their impact, and this research and subsequent toolkit can help museums to do this work.
Why Measure Museum Social Impact?
Here are a few ways you could use social impact data:
Share findings with funders, stakeholders, or legislators.
Use the data when applying for grants, to support new and existing programs or exhibitions at your museum, and to recruit board members.
Allow the data to inform strategic, master, and interpretive planning.
Share the findings internally, allowing them to shape marketing and public relations.
The national study is helping our team better understand how museums can successfully recruit visitors to participate in such studies. While some host museums are relying on regular visitors or lapsed members as study participants, other host museums are recruiting first-time visitors through a myriad of recruitment methods. We, and the host museums, are reflecting on not only how this type of study can be used to recruit new visitors to the museum, but also how to turn those new visitors into regular visitors or even members. We are hopeful that the toolkit will include strategies for audience outreach and development.
Resources
To learn more, follow updates, and see the 38 participating host museums, visit museumsocialimpact.org
Kari Ross Nelson, Jennifer Ortiz, Stephen Ashton, and Emily Johnson, “Two Approaches to Measuring Social Impact in Utah Museums,” Curator: The Museum Journal, 2021
https://doi.org/10.1111/cura.12457
Comments