Using Role-Playing Games to Teach Astronomy: An Evaluation

by Paul Francis
The Australian National University
Received: 07/20/05, Posted: 10/24/05

The Astronomy Education Review, Issue 2, Volume 4:1-9, 2006

Abstract

Since 1998, I've been experimenting with the use of role-playing games to teach astronomy. Students play the role of competing teams of researchers, racing to solve some astrophysical mystery. In this article, I review what has been learned from using these games around the world over the last eight years. The most common problem encountered is a tendency for students to become overly political. An unexpected benefit of these games is the boost that they give to student self- confidence. Overall, they seem to work well with a wide range of students, ranging from ninth grade to graduate school, and students exposed to this game comment repeatedly on how the games changed their attitudes toward the scientific process.

1. INTRODUCTION

The basic idea of classroom role-playing games is this: instead of telling the students about a given topic in the normal lecture mode, they work it out for themselves, playing the role of competing teams of researchers. The class is divided into teams, and each team is given a briefing paper containing some clues about the chosen topic (and several red herrings). The teams wander around the lecture hall, exchanging clues until they can put together a complete theory. They then present this to the rest of the class, and if their peers like their theory, they win a small prize.

In the last seven years, more than 5,000 students have used these role-playing games at over 30 schools and universities around the world. In this article, I analyze how they have worked and show some surprising clues as to why they can be effective and how they can fail.

The motivation for using role-playing games originally came from a survey of first-year university physics students in Australia. This survey, carried out by Mario Zadnik from Curtin University, showed that many students perceived physics to consist of rote learning and to be dull and unsocial. We all know that this is the very opposite of research. I wanted to see if I could expose even first-year students to some of the excitement of putting together theories. I was inspired by many previous examples of innovative teaching, particularly peer instruction (Mazur 1997), interactive quizzes (e.g., Byrd & Coleman 2004; Zeilik & Morris 2004), and constructivist theories of learning (e.g., Kavanagh, Agan, & Sneider 2005; Sebastià & Torregrosa 2005). Role-playing games are widely used in business education and in university subjects such as environmental sciences, where they allow students to explore competing points of view. As far as I know, role-playing games are new to the pure physical sciences.

In this article, I briefly review the technique of classroom role-playing games (section 2). My research methodology is described in section 3. In section 4, I describe the ways in which these games can fail, and in section 5, I show a somewhat surprising factor that makes these games effective. In section 6, I briefly comment on the sociology of how the use of these games has spread, before concluding in section 7.

2. HOW THE ROLE-PLAYING GAMES WORK

I will illustrate how one of these role-playing games works using an example: a game designed to teach about the runaway greenhouse effect on Venus, which I use with first-year students at The Australian National University (a class of around 60 students, roughly equivalent to an Astro 101 class in U.S. terminology). A copy of this game can be found on my Web site (http://www.mso.anu.edu.au/~pfrancis/roleplay.html). The game is run in a normal lecture hall during a 50-minute lecture slot and occurs midway through a three-week segment on planetary sciences.

I start the lecture by telling the students that they are about to take part in a role-playing exercise:

"I'd like you to imagine that you are a group of internationally renowned scientists brought here at great expense by NASA to help them solve a perplexing puzzle: Why is Venus so much hotter than it should be? I'd like you to divide yourself into groups of around three people. Each group should send someone down to the front to pick up a briefing paper from me."

I hand out the briefing papers, typically about eight in number. In this case, one group plays the role of military researchers who are working on how to spot enemy spacecraft, and hence are experts in thermal balance and blackbody radiation. Another group works for an environmental charity that monitors pollution, so they know about the infrared absorption of various compounds. A third group builds spa baths and knows about evaporation rates and photo dissociation of water exposed to UV. Thus, none of the briefing papers says anything directly about Venus, but between them, students have all the facts needed to understand the runaway greenhouse effect.

"I'd now like each group to read over your briefing papers. You will find that each of you is an expert in some bit of science. At first it may seem that your expertise has nothing whatsoever to do with Venus. But in every briefing paper are some crucial clues to working out why Venus is so hot, as well as several red herrings. Just as in the real world, no single expert could solve such a problem. You will need to wander around the room and swap clues with other groups. Only by putting lots of clues together and working out which red herrings to ignore will you be able to figure out this mystery.

"Once you think you've worked out a good explanation of why Venus is so hot, you should come down to the front and present your theory to the rest of the class. Just as in the real world, it doesn't matter whether you get the answer right—what matters is getting an answer that everyone else will accept! If everyone thinks that your answer is pretty good, you will win a prize!"

I typically use glow-in-the-dark stars or small chocolates as prizes. The normal reaction of students to this introduction is stunned silence. But slowly they begin reading their briefing papers and discussing them among themselves. Students are often reluctant to start wandering around the room and swapping information, but once one or two groups start doing this, it snowballs, and soon the room is abuzz with activity. I wander around the lecture hall, listening in and giving occasional hints. This turned out to be one of the major benefits of these exercises; by listening in, I can often identify student misconceptions that I would never have picked up until it was too late had I been giving a traditional lecture.

The atmosphere in class is wonderful—excited students debating the science in an animated way, swapping clues, and thinking deeply about possible alternative theories. It may look like anarchy—I once had a security guard come in and try to eject the students, thinking that a riot was in progress—but, somewhat to my surprise, students do stay focused on the science and typically make rapid progress. Toward the end, I usually need to nudge one or two groups into making their presentations. The first group typically gets quite confused, and the other students shoot down their theories quite quickly. But typically, the second or third group to make a presentation comes up with something that the class will accept.

This is only a brief account. For more details of how these games work and some pitfalls to be aware of, see my original paper on the subject (Francis & Byrne 1999; it is also available on my Web site: http://www.mso.anu.edu.au/~pfrancis/roleplay.html).

3. RESEARCH METHODOLOGY

I have used three main methods to gauge how and why these games work or fail, two deliberate and one accidental.

3.1 Feedback from Other Users

I post copies of all role-playing games on the Web. Anyone is free to use and modify them, but I do request that users let me know how they worked. To date, I have received correspondence from over 30 users. Slightly over half are from various Australian and American universities, ranging from small liberal arts colleges to prestigious research universities. Eight users are high school teachers in Australia, the United States, the United Kingdom, and Seychelles, in addition to users from universities in Slovakia, Canada, and South Africa. The exercises have been used with age groups ranging from year nine (ninth grade in U.S. terminology) through to graduate school, the bulk being senior high school and first-year university students. Class sizes range from 8 to 150.

3.2 Focus Groups

In 2000, I carried out a research project about student perceptions of these role-playing exercises. As far as I could tell as lecturer, these exercises worked very well and were very popular with students. I was worried, however, that I might have been misled by a voluble subset of students and that there may have been a significant number of quiet students who resented these games. To test this, I collaborated with staff at The Australian National University's Centre for Educational Development and Academic Methods (CEDAM).

Together, we carried out a focus group analysis of student perceptions of these games. We used students in an introductory no-math astronomy class, very similar to an Astro 101 class in the United States. The class had an enrollment of 50, split equally between men and women. Most were science students but primarily not in the physical sciences. The focus group sessions were held toward the end of semester, after the students had done six different role-playing exercises, and were spread over six weeks of lectures and tutorials. The focus group discussions were carried out in tutorials. To avoid self-selection, no prior warning was given to students that the discussions would take place. Over 80% of students attending the class took part.

The focus group sessions were run by Dr. Chris Trevitt from CEDAM; I was not present. He was instructed to try and prize out any hidden dissatisfaction with these exercises but also to give the students the freedom to say what their perceptions of these exercises were. Notes were taken during the session and the conversation was recorded, but to minimize the chance of students keeping quiet about problems for fear of retribution, this record was not made available to me until after the course was finished and all grades finalized.

3.3 Minute Papers

I regularly use the technique of minute papers to monitor how students are learning in my courses. Minute papers are widely used across many academic disciplines to assess student understanding. At the end of the last lecture each week, I finish five minutes early and ask the students to write down on a scrap of paper the answers to two questions:

What is the most important thing you learned this week?
What is the most important unanswered question that this week left you with?

I collect and collate the information on all these scraps and present a digest back to the students at the start of the first lecture of the next week. They were intended to monitor whether students understood the course materials. If students basically understood the course materials, the "unanswered question" responses would be diverse. But if I'd explained something badly, nearly all minute papers would mention this topic, allowing me to revisit it rapidly and fix the problem.

I did not use minute papers with the intention of gauging the effectiveness of role-playing exercises, but they proved to be an unexpectedly rich mine of information on how students perceived these exercises and what they were learning from them.

4. HOW THESE GAMES CAN GO WRONG

In this section, I discuss failure modes of these exercises. In particular, there was one problem that I expected to happen that never did, and one unexpected problem that seemed to occur regularly.

When I first tried using role-playing games, my main worry was that the class would get out of control and start gossiping, throwing paper airplanes, or chatting about the cricket. Somewhat to my surprise, I've never encountered this problem. Students are, if anything, much more focused while doing these exercises than they are in conventional lectures. I was worried, however, that other instructors with different student populations might find it a problem.

I found no evidence of this. Indeed, several users commented that it got students involved who were not normally engaged by lectures. Here, for example, are comments from teachers at two U.S. colleges:

"One hard-core cynical student who seldom does anything was sitting by himself looking through the book and I thought he was not participating, but as it turns out, he was trying hard to find the answer to a question that the rotation group could not answer using his textbook, and we had a really nice discussion about the question (and it was the first time he showed any interest this semester."

"The exercise went quite well. The students were much more willing to participate than I would have thought given that many of them are in the class only to satisfy a distribution requirement, and so they don't always have a very positive attitude."

The unexpected problem first surfaced when one of my games was being used with students as a prestigious U.S. research university. The class became highly political, and instead of sharing information, they started deliberately misleading each other and withholding information so as to gain a competitive advantage. I have since seen this same behavior in one of my own classes (a group of year 9th- and 10th-grade students) and have received two other accounts of the same problem. Here is an example:

"They didn't come to a full resolution because there was too much lying going on! I mentioned the possibility of concealing information or misleading people and they took it to heart. As a result they were struggling a bit to separate the facts."

This problem has shown up in 3/5 of reports from instructors using these games with students in grades 9 and 10, but only 1/30 of reports with older students. Especially with younger students, it may thus be prudent not to play up the competitive nature of these games too much in the initial briefing.

Several other problems were also mentioned, though none as frequently as this politics problem. Most were anticipated and addressed in my original paper (Francis & Byrne 1999). Two rather interesting new problems were reported, however, albeit by only one instructor each. The first problem was noted with a group of Australian high school students who were doing an extension course in cosmology:

"Most interesting was that a majority of groups seemed insistent that they could work it out by themselves. I would stress the point that they didn't have all the information and they should wander off and get some info from another group . . . and then attempt to work that into THEIR knowledge base. They didn't readily catch the idea that the solution might come from GAINING knowledge and piecing that together. A few groups actively refused to get information from others. They worked on the problem and would approach me with theories based on ideas they'd heard or read about, things they had taught themselves in the past. The knowledge some of these kids had was amazing, but didn't come near to allowing them to get a full answer."

The second problem occurred with students at a U.S. liberal arts college:

"The only problem came when teams presented their theory. The first one had gaping holes and yet no one in the audience raised any objections or asked for clarifications. At this college, there is what has sometimes been termed "the culture of nice," which means that the students will not readily express disagreement with each other or challenge each other."

5. STUDENT PERCEPTIONS OF WHY THE GAMES WORK

There is a great deal of evidence suggesting that students enjoy these exercises and perceive them as valuable. Both formal questionnaire surveys and anecdotal evidence confirm this, as discussed in my original paper (Francis & Byrne 1999). The focus groups failed to identify any disenchanted students. But what specifically is it about the games that they enjoy?

One unexpected factor showed up repeatedly in the focus group discussions, the minute papers, and even comments from other instructors: Students seem to like these role-playing games because they feel powerful and knowledgeable. Here are a couple of illustrative student quotes:

"You had your own sorts of breakthroughs when you found out another piece of information, and it was like Oh Wow! This fits in with this, and now we know this, and . . . you actually felt quite intelligent."

"We enjoyed these tasks more because it gives us the sense that we're the first ones to discover these things, and it gives us a sense of pride in what we were doing, whereas if we read it out of a book we wouldn't get the same sense of pride."

Students noted that it was a wonderful feeling to be "expert" in a given topic (by dint of having read the briefing paper) and to have others coming to them asking for information. They felt a sense of pride in working out puzzles for themselves, even partially. Feeling intelligent in class was clearly a novel and exciting thing for many students!

I was not expecting this at all. I'm still not sure I fully understand it. It does seem that these exercises bolstered the self-esteem of a significant number of students. Because all student feedback was anonymous, it's impossible to know precisely which students respond in this way—I'm guessing the weaker students.

On reflection, I suspect that an astrophysics classroom is a pretty intimidating place for a mediocre student. Distant lecturers, clearly far more intelligent than you are, tell you stories of great researchers and the profound things that they discovered. The idea that there could be anything in common between them and you seems laughable. You struggle with assignments and get mediocre results, further confirming your lowly place in the intellectual class system.

I find it very heartening that these role-playing games can help break down this gap between the students and the researchers about whom they learn. The self-help literature is full of accounts of how role-playing oneself in a successful situation can help overcome negative thoughts. But this was a purely accidental side effect of these games and not foreseen when they were written. If we can really understand this effect, deliberately crafted pedagogies may be able to do much better.

Another surprise: I usually hand out small prizes (plastic glow-in-the dark stars, for example) to groups brave enough to propose a theory to the rest of the class at the end of the role-play. This was intended to be just for fun. But many students mentioned that they really valued it. They fully realized that it was just for fun, but they thought that it was really important. They were not very specific about why it was valuable, but this message was repeated enough that it must have been telling us something important. I do not understand the psychology behind this either.

One final trend showed up in the minute papers. I had expected these to focus on astronomical details. But during weeks that I ran the role-playing games, over 40% of student responses to the question "what was the most important thing you learned this week?" were concerned not with particular astronomical facts but with more general issues, as illustrated by the following student quotes:

"The way that as a group of people, we as humans try and answer the way things are using a collection of seemingly unrelated facts."

"There are no right answers, only theories based on observations."

"Astronomy isn't all stiff attitudes and boring theories."

"The best way to learn is to ask questions. Never accept an answer or a theory without thinking it through, and if you disagree, without argument."

These answers had not been stated as learning goals of the course, so I was surprised that such a large fraction of students gave these answers. These general attitudes toward the scientific method are highlighted as important in recent discussions of what should be in introductory astronomy courses (e.g., Partridge & Greenstein 2003).

6. SOCIOLOGICAL NOTES

I've found it very interesting watching how the use of these games has spread. In this section, I briefly mention some interesting things that I've observed about the process. This is not in any sense a serious study, but I hope that you will find these tidbits interesting.

What stops people from using these games? Two main themes came though in talking to users. One was fear:

"Being new to teaching, I wasn't brave enough to try it right away, but now I think I will, this week."

"I had tentatively planned to do it last winter, but chickened out as sometimes I get cold feet when trying to do something ‘different.’"

Another obstacle was an overstuffed curriculum. I know that this is a major problem with physics courses, but I'm somewhat surprised that it's an issue with introductory astronomy courses because they are not prerequisites for further courses. Surely it makes sense to teach a few topics properly rather than race through all of astronomy as a shallow compendium of facts (a controversial issue)? But it does seem to be a problem in practice, as the following quotes show.

"I am excited to try these exercises, and don't really see any downside to them . . . except that my lecture time is reduced, and, for a 101 course that is everything-you-need-to- know-about-astronomy-in-sixteen-weeks, that could present a problem. But, hey, I'm going to try it. And, in the meantime, I am trying to talk my college into making Astronomy 101 into two semesters."

"I have had mixed success: I like a lot about them, but it takes a lot of class time."

Another surprise: when I originally put these exercises on the Web, I'd imagined that people would take the idea and write their own exercises adapted to their particular courses. Instead, over 80% of users took my games and ran them unmodified. Fewer than 20% modified them, and I only heard of one case in which new exercises were written.

7. CONCLUSIONS

Overall, I've been very pleased with the results presented in this article. These games seem useful, and the problems identified are not showstoppers. The effect that these games have on student self-esteem is perhaps the most interesting—and to me, surprising—result that came out, and one most worthy of further study.

Acknowledgments

I would like to thank the staff at CEDAM for their help in carrying out this research, the Department of Physics for being so supportive, the many users of these exercises around the world for sharing their experiences, and most of all, the students, who have been such willing guinea pigs in all this experimentation.

Resources

Copies of all my games and my previous paper about them are posted at http://www.mso.anu.edu.au/~pfrancis/roleplay.html.