This recording is from the Future of Museums Summit held October 29–30, 2024. In this session, Barry Joseph, Co-Founder, Brooklyn Seltzer Museum, explored the intersection of emerging AI technology, playful visitor experiences, and digital game design.

Transcript

Barry Joseph:

Hello. Jeff, I know what to do here. I got this. Yeah, loving these tools. Is there a way for me to see Jeff, number of folks here? Thank you for your patience as I get this started, people in session tab. There we go. So I see the tab, but I don’t see numbers. So if you can let me know how I can see numbers for planning some of the later stuff, that’d be helpful. Well now, we’ll get started. Welcome everybody to Shall We Play a Game? AI’s role in the Future of Museums.

Before we get started, I want to thank Jeff, who’s this room’s producer and [inaudible 00:01:36], the staff member who’s helping out, and of course everybody at AAM who’s making the future of museums summit happen. Super, super cool. I am, if you don’t know me, Barry Joseph, there’s a few ways you might’ve come across my work. I wear a few hats. I worked for six years at the American Museum of Natural History and last year AAM published my book, Making Dinosaurs Dance: A Toolkit for Digital Design in Museums.

I’m an adjunct faculty in learning technology and experience design at New York University. I recently founded the Brooklyn Seltzer Museum. I hope you come visit us. And I’m a digital experience design consultant who often works with museums. Last winter, for example, I led the team creating Video Games, The Great Connector. I’m also currently designing a board game and writing a piece of interactive fiction about how families use museums to make memories. And finally, yes, I’ve been teaching educators about how to use AI now for about two years. I’m now hoping that it’s the longest I’ll be speaking didactically. I think we should start playing a game because that’s what today’s going to be about. It’s going to be learning about AI and it’s going to be about interacting with each other. So let’s get our fingers warmed up. Let’s go into the chat and let’s see. Why don’t you say, in the form of a bullet point, one way that you used or considered using AI in your museum? Just a few words.

We’ll take about 30 seconds or 60 seconds. That’s about the time we’re looking at. And I’m encouraging you to lead, there we go. Logo development, in the accounting function, outdoor interactive tours, keywording, database building, creating resources is great. Keep going, keep filling it in. There’s going to be times where I’m going to be talking. Please continue to use the chat space. Sometimes I’ll be looking, sometimes I’ll look after, but this is, of course, a community of us together and so feel free to use that space and keep responding to that prompt please, because I’m going to be asking you to type in a moment because now that I see that you can access the chat, as we need, that we can work together and play a game.

Okay, I’m going to start. I’m going to ask a question, peanut butter and, please respond in the chat. What came to mind? I see a bunch of things, but what I see most common is jelly. And that was the answer. That was the answer, I should say, that I had in mind, but I never actually explained the rules. So for those who said jelly or thought about it, I guess you could tell you were supposed to fill in the blank. Guessing not just a new word, but the most likely. Next word. Also, did you only guess one word? I think most of you, I see banana smoothie, but most of you just did one word. No, most people didn’t say like jelly and then sandwich, right? Or peanut butter and jelly is a great sandwich to have on a picnic, right? You could have, but you didn’t. You had a sense that it was putting in one word and just filling it in.

All right, we’re going to do another round. Is there anybody who has a birthday this week? So put your name in and just say I’m birthday person in the chat. Birthday in October. There we go. So Kalani, this is all going to be for Kalani everybody. We call Kalani. Happy birthday to … Michelle wrote Kalani. A bunch of folks wrote Kalani. Some of you wrote you, you weren’t sure which one to do, right? We’re used to, in general, it’s probably just you, but I added something in addition. I added some context. This is Elizabeth Merritt. Kalani, sorry, I didn’t have a photo of you. But when you had context, Kalani, you put her name in. And that’s another rule you intuited without me explaining, that it wasn’t just the next word because you could have seen Kalani’s name and put in their last name, but it’s also the context.

So without me telling you, you changed the rule from most likely single word to the most likely single word in context. And these rules in general is how emerging AIs like Chatbot and others work, guess the single next word in context. Our new AI tools can produce so many remarkable and complex things. You can go back to the list of all the things you all just wrote at the top. These things can seem so much more powerful than just these three simple rules. Guess the next single word in context. In the end, that’s all that it comes down to. So at its core, understanding emerging AI, the tool we’re going to play with today to understand it better, can be as easy as one, two … three. Did you guess that? That’s, again, this next single word in context.

So we humans can be easily distracted by the promise that computers can be taught to think like us. But if we humans want to protect ourselves, and if museums want to prepare our visitors, we need to learn how to think like AIs. So who’s up for another game? I’m watching in the chat. [inaudible 00:07:04]. All righty, we’re going to play together. [inaudible 00:07:14]. All right. So we’re going to play something called Contexto. I don’t know if anyone’s played it before. I’m going to get to the bottom of the chat so I can see what we’re doing. So this is a game created with an AI. The AI uses what is known as a large language model by guessing the likely next word, creating an illusion of understanding.

So I need someone just to guess a word, any word. I haven’t explained the rules. So you can say anything. Anything appropriate, of course, it’s a family show. I’m watching in the chat for the first word someone’s going to type in. Museum. Excellent choice. So I’m going to type in the word museum and we’re going to see what’s going to happen. I typed in museum, and it said museum is number 14,663. We don’t know what that means yet. Now what I’m going to explain is this is telling us that we are just over 14,000 spaces away from today’s word of the day that the AI has selected.

Now, this is not Wordle. This is about a word’s meaning. And so we’re not looking for synonyms. We’re looking for words that most often might appear with the word we’re looking for. So I once guessed budget and then diet. And the secret word was calculator. Not because budget and diet are connected, but you might find budget calculator in the newspaper or diet calculator. So let me get another word. I see history. Let’s see if history brings us closer or further away from our word of the day. 10,000. We got closer. Who wants to guess the next word? Go ahead and type it in. So we saw a museum was 14,000, then history was 10,000. Bee. We’re going to try the word bee. It’s the first one I saw. Closer or further away. Wow, great job. We now got to 83. Went from [inaudible 00:09:08] blue. Blue means we’re really close. [inaudible 00:09:10] said that? Elizabeth. Good job. All right, so now that we see that museum’s pretty far, history’s closer. Bee’s a big jump. Emily said apple, we’re going to try apple.

Apple, sorry, pushed us back to 5,000, so the wrong direction. All right, but you can see it resort. Bee is still at the top. Who wants to guess a new word with our new information? Pollen. I’ll try pollen. Luckily this is unlimited, although of course we’re limited ourselves by time. Pollen’s closer, 265. Pollen 265, bee 83. Michaela says comb. We’ll try comb. C-O-M-B, come down. Comb. You guessed it in six guesses. Congratulations, Michaela. Wow. I have to say I’ve done this for two years. Never guessed it during the time. That’s amazing. Now what I want to do is go back and take a look at a little bit under the hood. So if we say closest words, we’ll see the top 10. Now the word was comb but look at the other ones from two to 10. Hair, brush, scissors, razor, tweezer. None of these has anything to do with the words we put in, right?

I’m going to hit the X. We can go back to our words. Can we see it? Yeah. Bee, pollen, apple. We saw the words, and we made up a narrative ourselves. We created a pattern. But it turns out the pattern wasn’t really there because the way we were thinking is not how Contexto thinks. Contexto, it is figuring out words that are connected to each other and we’re trying to guess them, but that’s not what we did to solve it. Now Michaela, I don’t know how you did it. I’m making a guess here, so if I’m wrong, forgive me. But we need to learn how to think the way AI thinks.

So more specifically, [inaudible 00:11:05] can tell the difference between how we experience the world, how we look for patterns all the time, and sometimes get it right and sometimes don’t, and how computers model the world, we might just have fighting chance. In fact, I’m designing a game right now to help us do just that. I need five volunteers. Now you’ve been volunteering great by typing. We’re actually going to move you from the chat into the screen. You will not have to show your video, but you will be expected to do your audio. So if you want to volunteer for the next game, say I volunteer, and Jeff will move you over to join me temporarily for the next game. Now you’re not typing. Excellent. Christie, we’ll bring you over in a moment. See if we can get a few more volunteers. Great, get you over.

All right, so while you’re being brought over, I’m going to go back to sharing my deck. Deck. And so I’m asking [inaudible 00:12:30] to do two things now, bring you over and to put my slides up. There we go. Good. Let me jump ahead to where we are on the deck now. Great. There we go. All right, so I started playing with Midjourney in November 2022. If you don’t know Midjourney, it’s one of those really popular AI tools for creating images with AI. Since then, I’ve generated over 7,000 images. I have my view. Most of these images have been for a game I call Uncanny Valley, a card game designed to welcome our AI overlords.

By now, we’ve worked through four complete sets of themes and elements, each time revised my criteria. I’ve removed living artists, for example, and learn where I need to avoid cultural appropriation. Even though I find the Afrofuturism cards look amazing. I offered the deck to play testers. They came up with over a dozen games to play with the cards and each iteration of the cards got better and better. And when I say better, I don’t mean closer to perfect, I mean better in highlighting the gap between how we experience the world and how AIs model them, which is what the game is all about, developing the visual literacy required to tell the difference. All right, so I am ready for us to try it out and I want to check with Jeff and see if we have our volunteers. I see Christie here and I don’t know if we’ve, here comes Emily. Christie, where are you coming from today on the planet?

Christie Weininger:

The Rutherford B. Hayes Presidential Library & Museums in Fremont, Ohio.

Barry Joseph:

Wonderful. Thanks for joining us. And Emily, where are you coming in from?

Emily Williams:

Dallas, Texas, The Sixth Floor Museum at Dealey Plaza.

Barry Joseph:

Wonderful. And I see, I think someone named [inaudible 00:14:22] is about to join us. Oh, I see. How do I see, I’m clicking here. Ah, Estelle, where are you coming in from today?

Estelle Rand:

From Beverly, Massachusetts, but from the Peabody Essex Museum in Salem.

Barry Joseph:

Wonderful [inaudible 00:14:47] time of the year for you. And Linda?

Estelle Rand:

Yes. Yeah.

Linda High:

Hi, I’m Linda. I am joining from the Gates Foundation Discovery Center in Seattle, Washington.

Barry Joseph:

Great, welcome. Jeff. If other people are joining, they can just keep coming in and we’ll start one round [inaudible 00:15:06] round. All right, so we’re going to try this out. Now what I want to share first is there’s a QR code out there. If you want to get your own access to the decks, you go here, it goes to the Instagram page and the link [inaudible 00:15:15] the top says where you can fill in a form to get a PDF of all this. All right, contestants, here’s how this is going to work. You’re all playing against each other.

Each of the cards in this deck, and here’s just four of them, have a number of characteristics. And what’s magic about this deck is that each card shares one of these characteristics with another card, but none of the other characteristics can be the same. So in our first round you’ll see these four cards. So I gave them numbers so they’re easy to reference, one, two, three and four. Say the thing they have in common, then list those two numbers. So if let’s say you see something in common between one and two, say that thing and then one and two and just shout them out and let’s see how we do

Christie Weininger:

Transportation, all four.

Barry Joseph:

Which cards?

Emily Williams:

The ocean.

Barry Joseph:

Which cards? If you say a thing, you got to say the cards also, so we can look [inaudible 00:16:11]-

Emily Williams:

Ocean, one and two.

Christie Weininger:

One, two, three, four, have transportation.

Barry Joseph:

We’re going to look for ones that are just two. So if you see more than two, it’s not one of the categories.

Christie Weininger:

Two and three-

Linda High:

The cats, two and three.

Emily Williams:

Two and three, the cats, yeah.

Christie Weininger:

One and four is guitar.

Barry Joseph:

Yep, that’s right. Any others?

Christie Weininger:

Four is rain.

Estelle Rand:

[inaudible 00:16:41]-

Linda High:

Three and four are in cars.

Barry Joseph:

Three and fourth cars. Yep. You see mountains in one and where?

Estelle Rand:

Three. One and three.

Barry Joseph:

So let’s pause now. Great job. Excellent everyone. Let me see if there’s anyone else who’s joined us. Just the four of us? Great. So I’ve been using these cards to teach educators how to think about AI and at the same time, how we can use games to teach complex topics. Let me go the next one. I’m going to play another round and this time I’m going to help you out a bit. You can see you get this card that shows you the six categories. There’s two types of subjects, like a donut and a dog. There’s roles you might see; there’s settings, colors is also one of them and then the format that it’s in. All right. And same as before. Just go ahead and shout them out. And if you’re in the chat, you can also make your guesses as well.

Estelle Rand:

Owls two and four.

Christie Weininger:

One and two are bicycles.

Linda High:

Cats, three and four.

Christie Weininger:

One and four, beach.

Estelle Rand:

Two and four are owl.

Linda High:

Children, one and three.

Emily Williams:

Technology, one and four. Okay.

Estelle Rand:

Green, one and three.

Emily Williams:

Ballet in two and three.

Barry Joseph:

I wasn’t sure if I was getting the ballet one. That one, I think, is a hard one. It’s amazing, if you look at the ballerina in three and you look at the owl in two, the owl is kind of in the same position but totally in an uncanny way. That’s not how owls work. And even humans don’t have three legs. So let me pause a moment and say thank you all so much for playing and being volunteers. Jeff is going to bring you back into the chat, so give a round of applause to our volunteers. Thank you so much. And I’m also seeing in the chat, Michelle called out the bike in one and two and saw the cats in three and four. So let’s look at that owl for a moment. If I was using AI to generate some art of an owl on a bike that was in the style of stained-glass, I would probably not use this. It’s pretty bizarre, but that’s the point.

It made sense to Midjourney. Some of you might know Matt Tarr, who works at the American Museum of Natural History, and he said something to me last week that I thought was really insightful. We talk about the idea that AIs have hallucinations, that they imagine things, but his point was they don’t imagine them anymore on the stuff they don’t imagine because they just can’t tell the difference to the AI Midjourney, that owl makes as much sense as the ballerina [inaudible 00:19:05] in number three. So the point of these cards, which again you can use the QR code to get free copies from me, is to be able to help people looking at them, start noticing the things that look off. Because if we don’t have that visual literacy for visual things produced by AI, for audio produced by AI, for videos produced by AI, for text produced by AI, we won’t be able to see through them and they’re just going to get more and more sophisticated.

Alrighty, we’re now going to move to our next activity, which is going to take the bulk of our time together, our next 40 minutes. So we’re going to do some museum mapping. We’re going to look specifically now at AI in museums. And to do this, I have this grid here. And so you can see there’s two lines. The line that goes from bottom to top is the disruption level. Things at the bottom are trivial in the impact they’re having. They’re almost having no impact at all. If it’s at the top of the line, that’s a game changer. That’s massive. That’s changing everything. The horizontal line is the BS level. The all the on the left, that’s a hallucination. There should be no concern about this at all. It’s completely made up. It’s nonsense. All way on the right is, oh this is real, we got to deal with it.

So we have the vertical, which is how disruptive is the thing we’re going to talk about. And the horizontal is how much it’s real or not. So let’s work with an example before we use a real one. So for those who were around 20 years ago. 20 years ago? 30 years ago, streaming television, you know what, I’m sorry, I don’t mean to jump back that far. 10 years ago, streaming television, what was it going to mean? The Netflix and the Amazon Primes. And so we start with end of live TV. That was a hallucination. It really wasn’t something that was real. It didn’t end live TV, but if it did, it’d be a game changer, so it goes in the upper left-hand corner to the right. We have breaks creative business model. That is a game changer. It’s completely transformed the way cable television works and it is quite real.

You go to cable model fails, that’s what it really broke. And so it’s even more of a game changer, so that went higher. If we go down to the lower left, we have no more water cooler. This concern that no one was going to have anything to talk about, we weren’t going to have any shared common culture anymore. Well that turned out to be hallucination, didn’t really happen at the same way. And even if it did, it’s somewhat trivial. And then at the bottom, end of the remote control. It didn’t end the remote control and not that big a deal. So you might not agree with where I placed them, but that’s all right because you’re going to get to do that because what we’re going to do now is apply this, not to streaming television, but generative AI in museums. And you’re going to work in teams of four, so that means you’ll be paired with three other people, and you’ll be working on a Miro board.

If you are doing this from your phone because you’re calling in, you want to start, go to that QR code right now. So if you don’t have Miro, an app on your phone, this part’s going to be rather difficult. Let me just say, it’ll be impossible. But otherwise, you don’t need Miro account. If I set this up properly, with some people who help me out, you’ll be able to go to it in a moment, but don’t go yet. So what I’m going to do is I’m going to go through the steps on the left and the first one [inaudible 00:22:15] go to the Miro board. Again, don’t go yet. The link to it is in the chat. Jeff just put it in. So when I say go, you’re going to go and I’m going to go through the steps now and then you’re going to forget them and that’s fine.

I’m going to leave this screen up and you’ll come back, and the steps will be here. But I want to walk through them now and see if you have any questions. So first, number one, you’re going to go to the board. Number two, you’re going to find the board with your number. So if I’m in group 14, I’m going to look for the group with the 14 in the middle. And right now, you can see there’s an example of it. In group 14, step three, you’re going to see there’s a number of colors, orange, blue, green and purple. You pick one, there’s only three other people in your group. You should be able to pick pretty fast. And then number four, everyone at the same time, without talking, will take one minute to separately write on each of your three stickies one and only one way you see people connected to museums thinking AI will impact your world. One and only one way you see people connected to museums thinking AI will impact the world.

You don’t remember the question; it will say it if you zoom out on the Miro board or come back here. Then, here’s where it gets fun. Number five, once you’re ready, the orange and blue pair are going to go first. They’re going to show their six stickies to each other. If you are really a competitive person, you can cover your stickies with another sticky and then reveal them. It’s up to you, time. But again, you have a minute to do it, to put it together, and then you can compare your six. If you give them match because they’re kind of similar or even the same, add to it whatever words you wrote, add the number two. You’re going to give it a power of two now.

All the ones you didn’t match, don’t worry about that. Then step six, switch. Now whoever’s green and purple’s turn. Do the same thing. If any of your six match or similar enough that you figure, yeah, it’s kind of the same thing, give that one a point value of two. By the end of step six, you’ll have a whole bunch of stickies. Some, you’ll have given two to. I forgot to say if you gave one a two, it means it’s a pair. Delete the one that’s redundant, you only need one of it. So if you and I have the same one and you’re going to put a two on yours, I’ll just delete mine. So the end of step six, you have a bunch of cards that have a number two on it and a bunch of don’t have anything.

Now in step seven, here’s what you’re going to do. You’re going to compare all the ones that remain and if any match, you’ll add their numbers together. So it might be orange, and blue might’ve had one which had no matching and [inaudible 00:24:35] might’ve had one with purple that had two. You’ll combine them. One of those stickies will now get a three and you’ll delete one of them. Or maybe you each had two that were the same, that powered up. So now two and a two will become a four. So you just add them up. After you do that, make sure you delete all the redundancies and put ones in all the ones that don’t have anything. So by the end of step seven you have a whole bunch of stickies that will have a leveled up from one to four, how common your thoughts were around those topics.

And then step eight, go and place it on the matrix. So below all of the boards is a matrix that we made just for this. And you’ll see it’s that yellow, blue, purple and red. And it has the same words we just described. So go ahead and spend them about five minutes filling that out, maybe 10. All right, so you you’re going to have a 15 minutes total. Let me do math for a moment. I’m holding it here. 55, you’re going to have 12 minutes. I’m giving you 12 minutes. So in a moment, Jeff’s going to send you over. You can always come back here if you want and ask me some questions. I’ll be here for answering questions. Otherwise, I’ll be watching you in the Miro board. Otherwise, in 12 minutes, you’ll be pushed back here and I’m going to take a quick look at what you guys came up with in our community [inaudible 00:25:45].

[inaudible 00:25:45] if you’re still doing stuff around, you can, but we’re going to be talking here about it, in the minute or two that we have left. So I’m going to go ahead and bring up the Miro board. And again, if you’re still placing a few, well do that. But let me first see … hear me. Can someone in the text [inaudible 00:26:15] know if you can hear me? Testing, testing. Excellent, thanks Emily. So in the few moments we have left, I’m just going to highlight some of the things you’ve done, but you can keep working on it if you want and you have the link to it. You’ll see it when it’s done. So I’m going to be looking for things that are the twos. I already saw under game changer, in reality number two, loss of job. That came for at least two people. And I assume the other ones here are one. Here’s another two.

Content created based on specific visitor interests and content level. Really [inaudible 00:26:52] game changer in reality. So those two, loss of job and content for people’s needs. We have increased disinformation up as a big game changer and a [inaudible 00:27:02], organization of large databases, improved accessibility, inclusivity, operating visitor interactives and cut down time for some collections management processes. And then down almost towards the middle we have writing descriptions.

Now let’s move to the other side on hallucination, we have a big game changer, but a hallucination, AL will automize, I think it may be automate, [inaudible 00:27:26] processes. And we also have a little bit more on reality side, historical figures and questions, but meaning hallucination. You don’t think that’s going to happen. People are talking about it. Trivial side, but also reality, we have research. That’s pretty important to me. I’m curious to know why. Feel free to write in the chat why you wrote research and require human input by accurate descriptions. That’s happening, but that’s not as big of a game changer.

So, I want to see this one. All humans replaced. That is a hallucination, not going to happen in midway between game changer and trivial. So you can see this is an exercise to help us get a sense of the room, but more importantly to start a conversation. This is the beginning of a conversation we’re having now, not the end. And I hope the community here at Future of Museums will continue to have that conversation and it’s one you can bring back. So if you have a team, feel free to go back to your team, of course, and you can do this exercise. And I also wanted to mention, I highlighted that I had, and I have a discount code for my book about digital design in museums. Hit me up if you’re interested. And if you want to play test Exit Through the Gift Shop, my interactive fiction that I’m doing, hit me up as well. And otherwise, I want to thank you all so much for today and I hope you have a good rest of your Future … of Museum Summit. Thank you.