Learnlets
Clark Quinn's Learnings about Learning
(The Official Quinnovation blog)

Good formal learning consists of an engaging introductions, rich presentation of concepts, annotated examples, and meaningful practice, all aligned on cognitive skills. As we start seeing user-generated online c, publishers and online schools are feeling the pressure. Particularly as MOOCs come into play, with (decreasingly) government funded institutions giving online content and courses for free. Are we seeing the demise of for-profit institutions and publishers?

I will suggest that there’s one thing that is harder to get out of the user-generated content environment, and that’s meaningful practice. I recall hearing of, but haven’t yet seen, a seriously threatening repository of such. Yes, there are learning object repositories, but they’re not yet populated with a rich suite of contextualized practice.

Writing good assessments is hard. Principles of good practice include meaningful decisions, alternatives that represent reliable misconceptions, relevant contexts, believable dialog, and more. They must be aligned to the objectives, and ideally have an increasing level of challenge.

There are some technical issues as well. Extensions that are high value include problem generators and randomness in the order of options (challenging attempts to ‘game’ the assessment). A greater variety of response options for novelty isn’t bad either, and automarking is desirable for at least a subset of assessment.

I don’t want to preclude essays or other interpretive work like presentations or media content, and they are likely to require human evaluation, even with peer marking. Writing evaluation rubrics is also a challenge for untrained designers or experts.

While SMEs can write content and even examples (if they get pedagogical principles and are in touch with the underlying thinking, but writing good assessments is another area.

I’ve an inkling that writing meaningful assessments, particularly leveraging interactive technology like immersive simulation games, is an area where skills are still going to be needed. Aligning and evaluating the assessment, and providing scrutable justification for the assessment attributes (e.g. accreditation) is going to continue to be a role for some time.

We may need to move accreditation from knowledge to skills (a current problem in many accreditation bodies), but I think we need and can have a better process for determining, developing, and assessing certain core skills, and particularly so-called 21st century skills. I think there will continue to be a role for doing so, even if we make it possible to develop e necessary understanding in any way the learner chooses.

As is not unusual, I’m thinking out loud, so I welcome your thoughts and feedback.

On a recommendation, I’ve been reading Jonah Sach’s Winning the Story Wars.   While it’s ostensibly about marketing/advertising, which interests me not, I was intrigued by the possibilities to understand stories from a different perspective.   I was surprised to find that it offered much more.

The book does cover the history of advertising, going through some classic examples of old-style advertising, and using some surprisingly successful examples to elicit a new model.  Some personal stories and revelations make this more than a conceptual treatise.

The core premise is turning your customer into a potential hero of an important journey.  You play the role of the mentor, providing the magic aid for them to accomplish a goal that they know they need, but for a variety of reasons may have avoided.  The journey is motivated from core values, a feature that resonates nicely with my personal quest for using technology to facilitate wisdom.

The book also provides, as one of the benefits, a nice overview of story, particularly the hero’s journey as synthesized by Joseph Campbell across many cultures and time periods.  If you find Campbell a tough read, as many do, this is a nicely digested version.  It talks in sensible ways about the resistance, and trials, and ultimate confrontation.

The obvious focus is on new way to build your brand, tapping into higher purpose, not the more negative fears of inadequacy.  So this book is valuable for those looking to market in a higher way.  And I do intend to rethink the Quinnovation site as a consequence.  But I suggest there’s more.

The notion of the individual being offered the opportunity to play a transformative role seems to be a useful framing for learning. We can, and should, be putting learners in meaningful practice roles, and those roles can be coming from learners’ deep motivators. One of the heuristics in learning game design is Henry Jenkins’ “put the player in a role they’d like to be in”.  This provides a deeper grounding, put the learner in a role they aspire to be in.

I think this book provides not only practical marketing advice, but also guidance for personal journeys and learning.  I think that the perspective of designing stories and roles that are based on personal values to be a great opportunity to do better design. I haven’t completely finished it yet, but I’ve already found enough value in the majority of it to recommend it to you.

A colleague recently queried: “How would you support that Jeopardy type games (Quizzes, etc.) are not really games?”  And while I think I’ve discussed this before, I had a chance to noodle on it on a train trip.  I started diagramming, and came up with the following characterization.

I separated out two dimensions. The first  is differentiating between knowledge and skills.  I like how Van Merriënboer talks about the knowledge you need and the complex problems you apply that knowledge to.  Here I’m separating ‘having’ knowledge from ‘using’ knowledge, focusing on application.  And, no surprise, I’m very much on the side of using, or doing, not just knowing.

The second dimension is whether the learning is essentially very true to life, or exaggerated in some way.  Is it direct, or have we made some effort to make it engaging?

Now, for rote knowledge, if we’re contextualizing it, we’re making it more applied (e.g. moving to the skills side), so really what we have to do is use extrinsic motivation.  We gamify knowledge test (drill and kill) and make it into Jeopardy-style quiz shows.   And while that’s useful in very limited circumstances, it is not what we (should) mean by a game.  Flashy rote drill, using extrinsic motivation, is a fall-back, a tactic of last resort.  We can do better.

What we should mean by a game is to take practice scenarios and focus on ramping up the intrinsic motivation, tuning the scenario into a engaging experience.  We can use tools like exaggeration, humor, drama, and techniques from game design, literature, and more, to make that practice more meaningful.  We align it with the learners interests (and vice-versa), making the experience compelling.

Because, as the value chain suggests, tarting up rote knowledge (which is useful if that’s what we need, and sometimes it’s important, e.g. medical terminology) is better than not, but not near as valuable as real practice via scenarios, and even better if we tune it into a meaningful experience.  Too often we err on the side of knowledge instead of skills, because it’s easy, because we’re not getting what we need from the SME, because that’s what our tools do, etc, but we should be focusing on skills, because that’s what’s going to make a difference to our learners and ultimately our organizations.

What we should do is be focusing on better able to do, moving to the skill side. Tarted up quiz shows are not really games, they’re simplistic extrinsic response trainers.  Real, serious, games translate what Sid Maier said about games – “a series of interesting decisions” – into a meaningful experience: a series of important decisions.  Practicing those are what will make the difference you care about.

Attending the SolutionsFest at the Learning Solutions conference last week, despite my earlier rant, I saw signs of hope.  A quick spot check revealed a number of approaches going above and beyond.

One direction I was pleased to see was a move to more performance support. I saw several solutions that were more focused on providing information as needed, or letting you  navigate to it, rather than thinking everything had to be ‘in the head’. This is definitely a promising sign.  They’re not hard to build, either.

The second promising sign was the use of scenarios. Several different solutions were focused on putting learners into contexts and asking them to perform. This is definitely the direction we need to see more of.  And, again, it’s not that hard to do!

One interesting takeaway was that the innovative solutions seemed to come more from small or internal groups rather than the big teams.  Which only reinforces my overall concern with the industry as a whole.  I wonder if it’s easier for small teams to adapt to advice of folks like Michael Allen (no more boring elearning), Julie Dirksen (Design for How People Learn) and Will Thalheimer than it is for big teams, who not only have to change processes but also educate their customers.

This is an unscientific sample; I did a quick tour of the displays, but couldn’t see all as there were some that were just too crowded.  I also looked at them relatively briefly and didn’t make comprehensive notes, so this is just a read of my state of mind as I finished.  It doesn’t ameliorate the overall concern, but it does provide some hope that things are changing in small pockets.

Daniel Coyle gave a wonderfully funny, passionate, and poignant keynote, talking about what leads to top performance. Naturally, I was thrilled to hear him tout the principles that I suggest make games such a powerful learning environment: challenge, tight feedback, and large amounts of engaging practice. With compelling stories to illustrate his points, he balanced humor and emotional impact to sell a powerful plea for better learning.

I think Ruth Clark’s provides a great service in presenting what the research says on elearning, starting with her highly recommended book eLearning and the Science of Instruction.  So it’s hard to want to quibble, but she put out what I think is a somewhat irresponsible post on games with the provocative title “Why Games Don’t Teach“.  So it’s only fair that I raise my objections, though the comments do a great job also of pointing out the problem.

As many have pointed out, the title is needlessly confrontational.  It’s patently obvious games teach, simply by trying a popular game yourself and realizing quickly that there’s no way you’re going to achieve a competitive level of play without substantial practice.  As Raph Koster’s fun and valuable book A Theory of Fun for Game Design aptly points out, the reason games succeed is that they do require learning.

So the real point Ruth is making is that research doesn’t show the value of games for learning, and that there are no guidelines from research for design.  And yet she continues to be wrong.  As Karl Kapp (author of Gamification) points out in his thoughtful and comprehensive comment, there are quite a few studies demonstrating this (and further elaborates on a study Ruth cites, countering her point).  As far back as the 80′s, frankly, Lepper and Cordova had a study demonstrating improvement from a game version of a math practice application.  The evidence is there.

What’s more insidious, as Koreen Olbrish points out in her comment, is that the definition of learning is open.  Unfortunately, what Ruth’s talking about seems to be rote memorization, by and large.  And we do know that tarting up drill and kill makes it more palatable (although we need to be quite certain that the information does have to be ‘in the head’ rather than able to be ‘in the world’).  But I maintain that rote fact remembering isn’t what’s going to make an organizational successful, it’s making better decisions, and that’s where games will shine.

Games, properly used, are powerful tools for meaningful practice.  They’re not complete learning experiences, but next to mentored live practice, they’re the best bet going.  And principles for design?  Going further, I believe that there are sound principles for design (heck, I wrote a book about it). It starts with a laser focus on the objectives, and the important ways people go wrong, and then creating environments where exercising those skills, making just those decisions that learners need to be able to make, are made in a meaningful context.

Yes, it requires good design. And, essentially the same basics of good learning design as anywhere else, and more, not other.  The problem with research, and I welcome more and a taxonomy, is that research tries to whittle things down into minute elements, and games are inherently complex, as are the decisions they’re training.  There are long-term projects to design environments and conduct the small elements of research, but we’ve good principles now, and can and should use a design-based research approach.

Overall, I think that it’s safe to say that:

• games can and do teach
• we have good principles on how to design them
• and that more research wouldn’t be bad

However, I think the article really only makes the latter point, and I think that’s a disservice.  Your mileage may vary.

When I run my game design workshop, one of the things I advocate is exaggeration.  And you need to take this to heart in two ways: one is why it’s important, and the other is how to get away with it.  That is, if you want learning to be as effective as possible.

When individuals perform in the real world, they’re motivated.  There are consequences for failure, and rewards (tangible or not) for success.  Yet in the learning experience, if we try to make it as real as possible, those motivations aren’t likely to be there. We want it to be safe to fail, so there aren’t quite the same consequences.  How do we make it closer to the real context, to maximize transfer?  Exaggerate the circumstances:

• You’re not working on a patient, you’re working to save the ambassador’s daughter
• You’re not just making a deal, you’re making the deal that the business is going to need to keep from going into receivership
• You’re not just designing a networking solution, you’re designing one to support the communications for the rebels overthrowing the oppressive regime

You get the idea.  You have to avoid stereotypes, and it’s a delicate dance to make it something that the learners will not dismiss but buy as more interesting than if you played it straight. Still, it’s worth the effort.  And testing’s a good idea ;).

You want these exaggerations not just for task-based motivation, but you also want to make it more meaningful to the learner.  Tap into not only the right application of the decision in context, but also a context learners will care about.

How do you get away with this?  You know that you’re going to get reined in, stakeholders are always so precious about their content (“you can’t be flip about X!”).  So go further than you think you’ll get away with.  You’ll get reined in, but then you’ll still end up at a reasonable place.

For example, we were doing a learning experience about using web tools, and learners were going to create a web site.  If you want youth to do it, we could’ve gone in different directions: a site for some environmental need, or for a subversive organization.  In this case, we decided it would be plausible (larger context was a design agency) to do a client band.

If we’d said rock band, we’d be reined back to a pop band.  Instead, we chose Goth Polka (and were surprised to find that it exists), and had a band name of the Death Kloggerz (in appropriate font).  The client reined us in, because Death was too far for them to countenance, so we ended up with the Dark Kloggerz.  Still, we got to keep in the odd elements that we felt were appropriate to keep it from being too banal.

The stakeholders will rein you in, but fight for the extreme. Your learners will thank you.

I just reviewed a paper submitted to a journal (one way to stay in touch with the latest developments), and all along they were doing research on the cognitive and motivational relationships in the game. They claimed it was a game, and proceeded on that assumption.  And then the truth came out.

When designing and evaluating learning experiences, you really want to go beyond whether it’s effective or easy to use, and decide whether it’s engaging.  Yes, you absolutely need to test usability first (if there’s a problem with the learning outcomes, is it the pedagogy or the interaction?), and then learning effectiveness. But ultimately, if you want it optimally tuned for success, pitched at the optimal learning level using meaningful activities, it should feel like a game.  The business case is that the effectiveness will be optimized, and the tuning process to get there is less than you think (if you’re doing it right).  And the only real way to test it is subjectively: do the players think it’s a game.

If you create a learning experience and call it game, but your learners don’t think it is, you undermine their motivation and your credibility.  It can be relative (e.g. better than regular learning) as you might not have the resources to compete with commercial games, but it ought to be better than having to sit through a page turner, or you’ve failed.

There are systematic ways to design games that achieve both meaningful engagement and effective education practice. Heck, I wrote a whole book on the topic.  It’s not magic, and while it requires tuning, it’s doable. And, as I’ve stated before: you can’t say it’s a game, only your players can tell you that.

So here were these folks doing research on a ‘game’. The punchline: “students, who started playing the game with high enthusiasm, started complaining after a short while, ‘this is not a game’, and stopped gameplay”.  Fail.

Seriously, if you’re going to make a game, make it demonstrably fun. Or it’s not a game, whether you say so or not.

I was contacted for a research project, and asked a series of questions. Thought I’d document the answers here, too.

Q0. How many years have you been designing educational games?

Q1. Please walk us through your process for creating an educational game from concept to implementation. Please use one of your games as an example.

A long answer is the only option (it’s a big process).  Using a design framework of Analysis, Specification, Implementation, and Evaluation:

Analysis

For any educational task, you have to start by looking at what your design objective is: you need to document what folks should be able to do that they can’t do now. I argue that this is most importantly going to manifest as an ability to make better decisions, ones that the learner doesn’t reliably make now.  It’s complicated, because SMEs don’t always have access to how they do what they do, and you have to work hard.  This isn’t unusual to learning design, except perhaps the focus on skills.

Then, you need to know how folks go wrong; what are the reliable misconceptions. People don’t tend to make random mistakes (though there is some randomness in our architecture), but instead make mistakes based upon some wrong models.

You also need to know the consequences of those mistakes, as well as the consequence of the right answer. Decisions tend to travel in packs, and if you make this one wrong, you’re then likely to face that other one. You need to know what these are.  (And the probabilities associated with them).

In addition, you need to know the settings in which these decisions occur, as many as possible.

And you need to know what makes this task inherently interesting (it is).  Here’s where the SME is your friend, because they’re so passionate about this they’ve made it the subject of their expertise, find out what makes them find it interesting.

Specification

With this information, you address those aligned elements from effective education practice and engaging experiences.    You need to find a storyline that integrates what makes the task interesting with the settings in which the decisions occur.  I like a heuristic I heard from Henry Jenkins: “find a role the player would like to be in”. Exaggeration is a great tool here: e.g. you’d likely rather be working on the ambassador’s daughter than just another patient.

You need to make those misconceptions seductive to get challenge. You don’t want them getting it right unless they really know their stuff.

You need to handle adjusting the difficulty level up at an appropriate rate; you might have complications that don’t start until after they’ve mastered the interface.

You need to specify characters, dialog, rules that describe the relationships, variables that code the state of the game, a visual (and auditory) look and feel.  The UI expressed to the learner, and more.

You’ll need to specify what the ‘perspective’ of the player is in relation to the character.

Overall, you need to nail meaningfulness, novelty, and the cycle of action and feedback to really get this right.

Finally, you need to specify the metrics you’ll use to evaluate your creation. What will be the usability goals, educational outcomes, and engagement metrics that will define you’re done?

Implementation & Evaluation

I’m a design guy, so I don’t talk so much about implementation, and evaluation follows the above.  That said…

The tools change constantly, and it will vary by size and scope. The main thing here is that you will have to tune.  As Will Wright said, “tuning is 9/10ths of the work”.  Now that’s for a commercially viable game, but really, that’s a substantial realization compared to how complex the programming and media production is.

Tuning requires regular evaluation.  You’ll want to prototype in as low a fidelity as you can, so it’s easier to change.  Prototype, test, lather, rinse, repeat.  (Have ever 3 words ever sold more unnecessary product in human history?)

There’s much more, but this is a good first cut.

Q2. Describe your greatest success, challenge, failure.

My greatest success, at least the most personally rewarding in terms of feeling like making a contribution, is definitely the Quest game. When you’re making a game that can save kids’ lives, you’ve got to feel good about it. On no budget (we eventually got a little money to hire my honors student for a summer, and then some philanthropic money to do a real graphic treatment), we developed a game that helped kids who grow up without parents experience a bit of what it’s like to survive on your own (goal: talk to your counselors).  Interestingly, I subsequently got it ported to the web as a student project (as soon as I heard about CGI’s, the first web standard to support maintaining ‘state’, I realized it could run as a web game), and it still runs! As far as I know, BTW, it’s the first web-based serious game ever.

My greatest challenge was another game you can still play on the web.  We’d developed a ‘linear scenario’ game on project management for non-project-managers, and they liked it so much they then asked for a game to accompany it.  But we’d already accomplished the learning!  Still, we did it.  I made the game about just managing to cope with missing data, scope creep, and other PM issues, so engineers could a) understand why they should be glad there were project managers, and b) that they shouldn’t be jerks to work with.

Biggest failure that I recollect was a team brought together by a publisher to work with the lead author on a wildly successful book series.  There was a movie script writer who’d become a game designer, and me, and a very creative team. However, we had a real problem with the SME, who couldn’t get over the idea that the ‘game’ had to develop the concept without getting mired in the boring details of particular tools. We would get progress, and then generate a great concept, and we’d be reined back in to “but where’s the tool simulation”?  Unfortunately, the SME had ultimate control, not the creative team, and the continuing back and forth ultimately doomed the project.

Q3. When determining game play is avoiding violence an issue? Q4. Is accounting for gender an issue when creating games?

I answered these two questions together; I don’t shy away from controversy, and believe that you use the design that works for the audience and the learning objective.  I believe education trumps censorship.  I argued many years ago (when Doom was the GTA of the day) that you could get meaningful learning experiences out of the worst of the shoot-em-ups.  Not that I’d advocate it.  Same with gender.  Figure out what’s needed.

As a caveat, I don’t believe in gratuitous violence, sex, or gender issues, (Why is sex more taboo than violence? I don’t get it.) but I believe you need to address them when relevant in context. In ways that glorify people, not violence or intolerance.

Q5. How did you develop your creation process?

I went from ad h0c at the start to trying to find the best grounding for process possible.  Even as an undergrad I had received a background in learning, but as a grad student I pursued it with a vengeance (I looked at cognitive, behavioral, constructivist, ID, social, even machine learning looking for insight).  At the time, the HCI field was also looking at what made engaging experiences, and I pursued that too. The real integration happened when I looked systematically at design and creative processes: what worked and what didn’t.  Using the learning design process as a framework (since folks don’t tend to adopt new processes whole-cloth, but tend to modify their existing ones), I worked out what specifically was needed in addition to make the process work for (learning) game design.

Q6. How do you work? Individually? As a team? If so, how do you develop a team?

Euphemistically, I work however anyone wants.  I seldom really do individual, however, because I have no graphic design skills to speak of (much to my dismay, but a person’s got to know their limitations, to paraphrase the great sage Harry Calahan).  Also, I strongly believe you should source the full suite of talent a game design needs: writing, audio, graphic, programming, UI, learning design, etc.  Naturally, in the real world, you do the best you can (“oh, I can do a good enough job of writing, and you can probably do a good enough job of audio as well as the programming”).

Q7. Is there a recipe for success in this industry? If so what is it and what would you say your biggest lesson has been so far?

My short answer is two-fold. I immodestly think that you really have to understand the alignment between effective practice and engaging experience (there’re lots of bad examples that show why you can’t just shove game and instructional designers into a room and expect anything good). Second, you have to know how to work and play well with others.  Game design is a team sport.

And finally, you really, really, have to develop your creative side.  As I tell my workshop attendees: I’ve got bad news, you have a big job ahead of you; if you’re going to do good serious game design, you’re going to have to play more games, go to more amusement parks, read more novels, watch more movies. It’s a big ask, I know, an onerous task, but hey, you’re professionals.   But you also have to be willing to take risks. Much to m’lady’s dismay, I argue that I continue to have to crack bad jokes as practice to find out what works (that’s my story, and I’m sticking with it).

If you can get a handle on these three elements: understanding the alignment, able to convince people to work with you on it, and push the envelope, I reckon you can succeed. What do you reckon?

Karl Kapp’s written another book, this time on gamification, and I certainly liked his previous book with Tony O’Driscoll on Virtual Worlds.  This one’s got some great stuff in it too, and some other ideas that raise some hackles.

Let me get one of the quibbles out of the way at the start: I hate the title “The Gamification of Learning and Instruction” (to the point I previously wrote a post arguing instead for ‘engagification‘).  Karl makes it clear that he’s not on the trivial notion of gamification: “Gamification is not Badges, Points, and Rewards”.  My problem is that by just having the title, folks who don’t read the book will still point to it to justify doing the trivial stuff. I’d much rather he’d titled it something like “Beyond Gamification” or “Engagification” or “Serious Gamification” or something.  He can’t be blamed for people misusing the term, and even his book, but I still fret about the possible consequences.

With that caveat, I think there is a lot to like here.  Karl’s got the right perspective: “Serious games and gamification are both trying to solve a problem, motivate people, and promote learning using game-based thinking and techniques.”  He does a good job of laying out the core ideas, such as:

“Games based on this complex subject matter work, not because they include all the complexities, but precisely because they reduce the complexity and use broad generalizations to represent reality. The player is involved in an abstraction of events, ideas, and reality.”

I liked his chapter 2, as it does a good job of exploring the elements of games (though it’s not quite as categorical as I’d like ;).  He’s got pragmatic advices there, and lots of examples to help illustrate the possibilities.  He goes beyond serious games in a number of ways, talking about adding motivation factors for other things than making good decisions. I worry somewhat that folks might (and do) use the same things to get people to do things that they might not otherwise believe are good to do, and the ethical issues aren’t addressed too much, but again that’s not Karl’s point, as his many examples clearly show.

Chapters 7-9 are, to me, the most valuable from my point of view; how do you do game design (the focus of Engaging Learning). Chapter 7 talks about Applying Gamification to Problem Solving and helps explain how serious games provide deep practice. Chapter 8 maps gamification on to different learning domains such as declarative, procedural, affective, and more. There are valuable hints and tips here for other areas as well as the ones I think are most important.  And Chapter 9 provides valuable guidance about the design process itself.

I wish there was more discussion of how meaningful challenges for problem-solving will make fact based learning more relevant, rather than just gamifying it, but that’s not necessarily the role of this book.  I very much like this statement, however: “The gamification of learning cannot be a random afterthought. It needs to be carefully planned, well designed, and undertaken with a careful balance of game, pedagogy, and simulation.”  Exactly!  You can’t just put instructional designers and game designers in the room together and expect good things to happen (look at all the bad examples of edutainment out there); you have to understand the alignment.

There are some interesting additional chapters.  Guest authors come in and write on motivations and achievements (Blair), the gamer perspective (presumably son Kapp), a case study of a serious initiative in gaming (Sanchez), and alternate reality games (Olbrish).  These provide valuable depth in a variety of ways; certainly Alicia Sanchez is walking the walk, and the alternate reality games that Koreen Olbrish are talking about have struck me as a really compelling opportunity.

There are flaws. I can’t comprehend how he can go from talking about objectives straight to talking about content.  Games are not about content, they’re about context; putting the player into a place where they have to make the decision that they need to be able to make as an outcome.  This statement really strikes me as wrong: “The goal of gamification is to take content that is typically presented as a lecture or an e-learning course and add game-based elements…”. Given my focus on ability, not content, this predictably irks me.

Karl also misses what I would consider are some important folks who probably should be referenced.  While he did get Raph Koster and Jane McGonigal, he hasn’t cited Aldrich, Gee, Shaffer, Barab, Jenkins, Squire, Steinkuehler, or even Quinn (ok, I had to say it).  It seems a bit narrow-focused to miss at least (the ‘other’ Clark) Aldrich, who’s written now 4 books on the topic.  I mean, being an academic and all…. :)

Overall, I know he’s fighting for the right things, and think there’s some very broad and useful information in here. If you’re looking to make your learning designs more effective, this book will show you a lot of examples, give you some valuable frameworks, and provide many hints and tips.