The quest has raged on and on: where’s the data on how effective games are? And the problem has continued: well, how do you evaluate the quality of the design of the game? Because, unless you feel confident the game is designed properly, you can’t decide whether a bad outcome (or even a good one) is due to the game, or something else. We have criteria for instructional design, but how can we compare?

I think this is an important issue that may be the biggest barrier we’ve had to trying to get the data people are demanding: real evaluations of games. There are other barriers: people doing evaluations but not wanting to publicize it as a competitive advantage, doing games but not evaluating them, but I’d argue that it’s hard to compare until you feel you’re comparing a well-designed game to a well-designed alternative. Clark Aldrich has done some good independent evaluation with Virtual Leader, and demonstrated improvements, but I’d like to see more on different scopes of games, in different domains, for a range of cognitive skills (and, as always, I’m not talking about tarted-up quiz show ‘frame games’ but meaningful cognitive decisions).

So, it occurred to me, the answer is in a framework for game design. Which, ahem, is what my whole book is based upon. I don’t know why I didn’t think of it before, frankly. And, before you accuse me of too much self-serving thinking, I do want to point out that I’ve been looking for other systematic frameworks for learning game design, and haven’t found them.

I’ve read Prensky, Aldrich, Gee, and now am on Shaffer, (and others, but these are the ones who’ve been writing specifically about learning games) and I see great stuff, but I haven’t seen what I can term a systematic design approach other than mine (again, I know how this sounds, but such a design approach was my very specific goal and opportunity). They all cover at least some elements of design, and I made an effort to review their approaches and make sure they didn’t have anything I didn’t at least explicitly consider.

And I’m happy to be wrong, but I have tried to be fairly exhaustive because I do care. And I’m sure there’s more richness that can be wrapped around what I’ve done (I’ve added some thoughts myself since the book came out), but I still think the core framework is sound and I’ve been looking at this for over 10 years (since my first article on the topic came out) and really more like 25 (when I first told my boss at DesignWare that we could be doing much more meaningful games than spelling drills).

So, what’s my point? I think that maybe what could be done and hasn’t been is to operationalize (a word I used to hate, but don’t have a better one to hand) my framework as an evaluation instrument as well as a design framework. It’s tough, because how do you evaluate how well the story integrated the decisions? Yet that’s what you have to come to grips with. It’s not something I can do in my copious spare time (independent, with children; what’s spare time?), but I think there’s an argument to be made that it’d be a useful contribution for someone to do. Ph.D. thesis, anyone?

Martine from Angils (a European-centric serious games group) asked me:

I just had a look at your blog and was interested in your views about virtual worlds and MMOs for selling learnlets…Many of the large service organizations I have spoken with are dealing with this type of proposition for some of their clients – where SL will be the test-bed for them to then develop their own virtual world for including the selling of learnlets.

My thoughts on learnlets originally were more that they could be viable commercially through websites, but certainly there’s no inherent barrier to them existing and being desirable in virtual worlds/MMOs. If one could provide a demonstrably effective and subjectively appealing experience for a skill in demand, there should be a potential transaction basis.

My thing, of course, is how to systematically design them to be effective and engaging. I’ve yet to find a better framework than the one I developed, but then I may be biased…;)

On the bigger scale, worlds for the sale of learnlets, I suppose it’s a virtual university with mini-courses. I’ve been trying hard to understand the value proposition for virtual worlds, as the overhead is high to get what I think are the unique contributions (e.g. co-creating models) but others are convincing me the personal aspect of building your own character and the social aspects are both ‘sticky’.

So, how do you build an interesting social life around the learnlets? Studying together, and learning together (learning can be more effective socially), so ways to find cohorts to do it together would be the selling point. And, of course, you’d need a way for people to connect and jointly experience meaningful and effective learning. You’d probably have different sizes (read: scopes) of the learning, and ideally you’d have different ‘styles’, different cohort sizes, etc.

One model would be a world just for this, another would be a way to integrate this into an existing world, whether 2nd Life, Entropia, or elsewhere where there’s an economy.   There clearly are ranges from paying someone else to do it (or purchasing the result) or learning to do it yourself.   Some of it naturally has to be available in the environment, but extensions or emergent capabilities could be a market.   It’s Pine & Gilmore’s ‘transformation’ economy (the last stage of the experience economy), virtually. Whether and how anyone locks it up is a different issue.

One of the recurrent ‘dreams’ is of stealth learning, where one could play a game and learn something without even being aware of it. It resurfaced again on the Serious Games mailing list, and somehow my thoughts finally coalesced. Here’s what I had to say:

I’ve often wondered whether stealth educational games are possible, and I had an epiphany yesterday when thinking about this (and was reminded of by Noah’s post today). In short, I don’t think there can be stealth education, at least in a reliable way. Let me explain why by analogy to analogy(!):

In classic analogical reasoning (the topic of my PhD thesis), Gick & Holyoak (1980) gave learners one experience with a problem, and then they were asked to solve an analogously related problem. The base rate of solution was low (e.g. 30%), unless externally prompted (then 75%). (Base rate without prior problem 10%). With two problems the likelihood goes higher, but not as effectively as if there’s guidance to explicitly abstract (Gick & Holyoak 1983).

My inference here is that presenting relevant problems without explicit discussion of transfer is not as likely to lead to the learning outcome as if you explicitly make the relation between the learning experience and the ‘real world’. In other words, your ‘stealth learning’ *might* work, but not as likely nor effectively as if you ensure some abstraction and explicit transfer to other similar problems.

Could you do that in a game in a way that it’s thematically consistent? Perhaps, but then you’re treading a mighty thin line between being explicit and being stealth.

So, guess I’ve convinced myself that it’s not plausible. Possible, yes. With enough opportunities to practice, and some embedded diagrams, you might well develop the capability. But without it being explicit, you might miss some opportunities to apply it. I fear it just wouldn’t be as robust as making it explicit. Once you’re done with the game, you might not mind having it pointed out what a valuable skill you have obtained (maybe like leadership in World of Warcraft).

Ernest Adams, one of my favorite writers about game design, has written a piece about mixing in action game elements into other formats. Along the way, he talks about accessibility, challenge, and design principles. Now, I’ve argued in different places for not throwing in gratuitous twitch elements, based upon some instinctive reactions, but he makes a cogent case why it’s wrong in a broader context.

In short, his argument is that throwing in twitch reactions rewards the twitch gamer, and precludes other players. And, when the game genre doesn’t naturally support it, it might preclude the audience you want. For learning games, this is more the case, when you want to keep the focus on important decisions. You can still have random events, keep score, etc, but don’t require motor skill coordination unless that’s essential to the ultimate performance.

Glad to have someone else say it, with the nuanced discrimination it needs. Recommended.

Last week we went down to LA to visit my Mother for her birthday, and to take the kids to Disneyland for a day (after the other trips we’ve taken for family reasons, including my Dad‘s rememberance). It was a great trip for all reasons, but the Disney experience had a lesson for me.

We had almost no lines the whole day even for top rides like the Matterhorn, Pirates of the Caribbean, Indiana Jones, etc. We caught Mr. Toad’s Wild Ride early, but late in the day went back to Fantasyland to hit Pinocchio, Snow White, & Peter Pan (none of which I’d been on in more years than I care to admit). We went on the first two, since the line for Peter Pan was substantially longer.   Finally we bit the bullet and got in line for Pan, and then I understood why people were waiting for it. Peter Pan was a substantially better ride, for important reasons.

Now, each of these rides has a ‘license’ (in the game industry, companies with ‘properties’ such as Lord of the Rings will license them to companies to make accompanying games, and no one else can make a competing game) they have to align with. The trick, then, is to make the ride a compelling experience in and of itself, as well as use the story associated with the license. With games, sometimes the experience *is* the story, that is you play James Bond in GoldenEye, and other times it’s another story with the same character (e.g. Indiana Jones and the Fate of Atlantis).

In a ride, the experience should be good on it’s own, whether or not you know the story. So, for example, our kids never got into the Pinocchio movie and consequently don’t know the story. Nor had they seen the Disney version of Peter Pan. Yet the Peter Pan ride was just very cool: you float out over the city in a sailing ship and into the stars, before coming down and flying around Neverland. While in Pinocchio, you basically just see the events in the story (with the one caveat of being eaten by the whale, which is scary). And Snow White didn’t even have a real ending, suddenly you’re just out!

Disney’s Imagineering has done amazing things, and those rides are old, but there’s a lesson here about getting the experience right, so that not only is the story referenced, but the rider actually has an interesting experience. That holds for learning game (er, Immersive Learning Simulation) design, too, where you don’t just want cognitive practice of important decisions, but you’d like the learner to be emotionally engaged. As I tell my workshop attendees, it’s not about designing content, it’s about creating an experience! So, think wholistically and create an environment that hooks you from the beginning, creates interesting emotional trajectories, and provides a feeling of closure at the end.

I was interviewed on simulation games by NexLearn (creators of SimWriter) for their April newsletter, and make a few points that I feel strongly about. Check it out!

They left out my claim that you can’t just put educators and game designers in a room and expect a viable solution.   Also that in the near future I don’t think we’ll see tuning to a game-level of experience as often as we might.   I also made a point that there are problems with choosing the tool before you do the design.   Finally, I said that you shouldn’t go it alone unless you have time and budget for experimentation, and that developing the skills to do this effectively take time.   But they left in some important points as well, and I think it’s worth a read.

This past week I was in Taiwan as an invited keynote at the IEEE’s Digital Game and Intelligent Toy Enhanced Learning workshop. I like to keep my head in on the academic side as a source of inspiration, and this was just such an opportunity. I got there late (family commitment meant I missed the first two days) but I heard the last day with some great talks, and had a chance to read a lot of the proceedings on the long flight back.

One of the interesting outcomes was the debate about what’s a game and what’s a toy. Games have rules, toys have affordances (read: capabilities), but when your toys can communicate to you and each other, they start blending the boundaries. Another form of blending was that some of the game work was about classroom work, but the toy stuff tended to be more focused on non-school play.

Of course, there was some talk about supporting the learning, and the need for reflection. In addition to my expected coverage of systematic learning game design, one of the points I tried to throw in is that we should be looking at ways in which learning systems could be smart about coaching learning to learn and generalization, not just on the particular domain such as mathematics. There’s very simple coaching in the Quest game that focuses on your exploration, based upon some work Valerie Shute & Jeffrey Bonar did even longer ago, and I think that model of coaching could be expanded and built into any modeled environment (e.g. game engines).

I didn’t hear his keynote, but Michael Eisenberg, who I’d met years ago and has subsequently become a steadfast innovator at the University of Colorado Boulder (a great cog sci place), had another talk about making magic manifest, not having black boxes but making the operations manipulable so you can change them and explore the underlying relationships. Eric Schweikardt, a student of Michael’s collaborator Dan Gross, attempted a categorization across games, and pointed out a different model of programming that involves lots of distributed capabilities being pulled together into a smart aggregation instead of a central intelligent program (e.g. Lego Mindstorms), and presented several versions.

My notion of a wise curriculum includes thinking systemically and modeling skills, so the notion of using toys to learn different modeling schemes is very cool. Not to the exclusion of the central control model, but as an alternate approach (indeed, as was pointed out by Schweikardt, Stephen Wolfram has argued that we should be using small rules as the way to understand how the world works).

Another innovator with toys was fellow keynoter Masanori Sugimoto who is doing some very innovative things with manipulables, including a computer projector. (I made a note to add ‘projector’ to my list of potential input/outputs for mobile devices!) He also does very systematic studies of his implementations and tunes them to get them better. For instance, he was using a camera to register what elements kids put down where on a grid table, but the kids leaning over obscured it, so he had to make the pieces carry the information and have the grid itself record what was on it.

As Professor Tak-Wai Chan (our host, and a recognized innovator in his own right for his exploration of intelligent learning ‘companions’) noted, one of the reasons to have this overseas is to help make the US aware of how much happens overseas; one of the first lessons I learned when I went to Australia for an academic position was how insular the US is, beacuse there’s so much happening in the US it’s easy to miss how much is happening elsewhere.

Sure, there were some fairly straightforward exercises about games and toys, and some rather typical research, but we need these too. The next one will be a full conference in Europe, and I believe there’s a commitment to regularly move it around. The neat thing about this conference was that it not only about classroom learning but also about informal learning (and technology, ok so I’m still a geek), so it provided an interesting way to look at the intersection, and I think there will be great reasons to keep track of this direction. There were a lot of students, and there’s great hope that this research (as eloquently put many years ago by John Anderson that we learn alot about learning by trying to create learning systems) can make new inroads into understanding.

Over at GamaSutra, there’s a great example of a Design Document for a game. As I talk about in my workshops, they freely switch between lookup tables, flow charts, and other ways of representing their thoughts.

It’s nice for them to be willing to share this peek inside commercial game development, and there’re some great ideas about things like different paths for different players (here: simple score, puzzles, and real ‘twitch’ skills). Recommended.