I was on a call yesterday with a team that had developed a stunning engine-driven scenario. It had rich complexity, good visual and interaction design, appropriate challenge, etc. This was something I’d be proud to show.

However, they’d had a focus group testing which had soundly rejected it. I was helping them try to understand was how could this happen.

What became clear was that a couple of simple, and avoidable, mistakes doomed the result. Mistakes that had little to do with the monumental task of creating the underlying model, and all to do with user perceptions.

First, the system threw you out the first mistake you made. And with no feedback about why! There were people who played it systematically to figure it out (your typical gamers), but the average audience was frustrated. This is easy to fix, by the way.

The second mistake was that they hadn’t set expectations for the focus group. The younger folks were unhappy it didn’t have the polish of a commercial game, and the older folks complained that the simulation didn’t have the depth of the real experience. Both of these could be addresssed by setting expectations up front (and the beginning information was typically dull and dry, not matching the dynamic music and going on way too long).

I suggested that with minimal work that focus group result could be turned around. A review before bringing it to the focus group would’ve caught this.

The end result was that the business decision was made to pull the plug on the project, and would regroup to think about whether to try again. It’s a shame, since they’d spent a reasonable amount of money (actually quite little for what they’d gotten), had a great core, and only needed to do a little window dressing. It’s also a lesson on just how important it is to sweat all the details.

The Association for Computing Machinery has been collecting recommendations for classic books in Computer Science, and it made me think of the key books that have influenced my thinking. This is part of a broader consideration of what books people doing learning technology should read.

I have an eclectic view, trying to take a broader view of technology supporting performance support, Knowledge Management, informal learning, etc, so I largely ignore traditional instructional design approaches like Gagne’ and Merrill. I’m a fan, note, and have read lots of Merrill for instance, but lets take that as a baseline and move beyond. (I recommend Reigeluth’s survey’s of instructional design approaches.)

So here’s a brief annotated list of readings I think are seminal:

• Don Norman’s Things That Make Us Smart gives us a valuable perspective on how to support people’s performance (everyone should’ve already read his Design of Everyday Things…).
• John Carroll’s Nurnberg Funnel, a landmark book touting a minimalist approach focusing on respecting (and leveraging) your learner’s real world knowledge.
• Allen Collins & John Seely Brown’s Cognitive Apprenticeship is not a whole book but either a chapter (with Susan Newman) or an article (with Ann Holum). In either case, it’s the best model for designing learning that I know, abstracting across work by Palincsar & Brown, Scardemalia & Bereiter, and Shoenfeld.
• Jeroen van Merrienboer’s Training Complex Cognitive Skills presents the best model for separating out knowledge from skills and addresses both.

In addition to these full thoughts, several other elements contribute to my design approach. Reigeluth’s Elaboration Theory talks elegantly about starting from a broad context and drilling down. Spiro’s Cognitive Flexibility Theory emphasizes the importance of multiple representations. Sweller’s Cognitive Load Theory helps us understand the importance of providing appropriate support in performance and even in representations. Mager’s specification of objectives helps us emphasize the resulting performance. And Keller’s ARCS model helps us incorporate the emotional side of learning.