Clark Quinn's Learnings about Learning
(The Official Quinnovation blog)

31 March 2010

Mea Culpa and Rethink on Pre-tests

Clark @ 1:24 PM

Well, it turns out I was wrong.  I like to believe it doesn’t happen very often, but I do have to acknowledge it when I am. Let me start from the worst, and then qualify it all over the place ;).

In the latest Scientific American Mind, there is an article on The Pluses of Getting It Wrong (first couple paragraphs available here). In short, people remember better if they first try to access knowledge that they don’t have, before they are presented with the to-be-learned knowledge.  That argues that pre-tests, which I previously claimed are learner-abusive, may have real learning benefits.  This result is new, but apparently real.  You empirically have better recall for knowledge if you tried to access it, even though you know you don’t have it.  My cognitive science-based explanation is that the search in some ways exercises appropriate associations that make the subsequent knowledge stick better.

Now, I could try to argue against the relevance of the phenomenon, as it’s focused on knowledge recovery which is not applied, and may still lead to ‘inert knowledge’ (where you may ‘know it’, but you don’t activate it in relevant situations).  However, it is plausible that this is true for application as well.  Roger Schank has argued that you have to fail before you can learn. (Certainly I reckon that’s true with overconfident learners ;). That is, if you try to solve a problem that you aren’t prepared for, the learning outcome may be better than if you don’t.  Yet I don’t think it’s useful to deny this result, and instead I want to think about what it might mean for still creating a non-aversive learner experience.

I still believe that giving learners a test they know they can’t pass at best seems to waste their time, and at worst may actually cause some negative affect like lack of self-esteem.  Obviously, we could and should let them know that we are doing this for the larger picture learning outcome.  But can we make the experience more ‘positive’ and engaging?

I think we can do more. I think we can put the mental ‘reach’ in the form of problem-based learning (this may explain the effectiveness of PBL), and ask learners to solve the problem. That is, put the ‘task’ in a context where the learner can both recognize the relevance of the problem and is interested in it.  Once learners recognize they can’t solve the problem, they’re motivated to learn the material.  And they should be better prepared mentally for the learning, according to this result. While it *is*, in a sense, a pre-test, it’s one that is connected to the world, is applied, and consequently is less aversive.  And, yes, you should still ensure that it is known that this is done to achieve a better outcome.

Now, I can’t guarantee that the results found for knowledge generalize to application, but I do know that, by and large, rote knowledge is not going to be the competitive edge for organizations.  So I’d rather err on the side of caution and have the learners do the mental ‘reach’ for the answer, but I do want it to be as close as possible to the reach they’ll do when they really are facing a problem.  If there is (and please, do ensure there really is, don’t just take the client’s or SME’s word for it), then you may want to take this approach for that knowledge too, but I’m (still) pushing for knowledge application, even in our pre-tests.

So, I think there’s a revision to the type of introduction you use to the content,  presenting the problem or type of problem they’ll be asked to solve later and encouraged to have an initial go at it before the concept, examples, etc are presented.  It’s a pre-test, but of a more meaningful and engaging kind.  Love to see any experimental investigation of this, by the way.

27 March 2010

The Great ADDIE Debate

Clark @ 11:34 AM

At the eLearning Guild’s Learning Solutions conference this week, Jean Marripodi convinced Steve Acheson and myself to host a debate on the viability of ADDIE in her ID Zone.  While both of us can see both sides of ADDIE, Steve uses it, so I was left to take the contrary (aligning well to my ‘genial malcontent’ nature).

This was not a serious debate, in the model of the Oxford Debating Society or anything, but instead we’d agreed that we were going to go for controversy and fun in equal measures.  This was about making an entertaining and informative event, not a scientific exploration.  And in that, I think we succeeded (you can review the tweet stream from attendees and some subsequent conversation).  Rather than recap the debate (Gina Minks has a short piece in her overall summary of the day), I’ll recap the points:

The pros:

  • ADDIE provides structured guidance for design
  • ADDIE includes a focus on implementation and evaluation
  • ADDIE serves as a valuable checklist to complement our idiosyncratic design habits

The cons:

  • ADDIE is inherently a waterfall model, and needs patching to accommodate iterative development and rapid prototyping
  • People use ADDIE too much as a crutch for design without taking responsibility for using it appropriately
  • It assumes courses

The pragmatics:

Steve showed how he does take responsibility, putting evaluation in the middle and using it more flexibly. He uses Dick & Carey’s model to start with, ensuring that a course is the right solution.  The fact that the initial ‘course, job aid, other problem’ analysis is not included, however, is a concern.

It also came out that having a process is a powerful argument against those who might try to press unreasonable production constraints on you.  If a VP wants it done in an unreasonable time frame, or doesn’t want to allow you to question the analysis that a course is needed, you have a push back (“it’s in our process”), particularly in a process organization.  You do want a process.

The Alternatives:

The obvious question came up about what would be used in place of ADDIE.  I believe that ADDIE as a checklist would be a nice accompaniment to both a more encompassing  and a more learning-centric approach.  For the former, I showed the HPT model as a representation of a design approach considering courses as part of a larger picture.  For the latter, I suggested that a focus on learning experience design would be appropriate.

Using an HPT-like approach first, to ensure that a course is the right solution, is necessary.  Then, I’d focus on working backwards from the needed change (Michael Allen talked about using sketches as lightweight prototypes at the conference, and first drawing the last activity the user engaged in) thinking about creating a learning experience that develops the learner’s capability.  Finally, I’d be inclined to use ADDIE as a checklist to ensure all the important components are considered, once I’d drafted an initial design (or several).  ADDIE certainly may be useful in taking that design forward, through development, implementation and evaluation.


I think ADDIE falls apart most in the initial analysis, not being broad enough, and in the design process: e.g. most ID processes neglect the emotional side of the equation, despite the availability of Keller’s ARCS model (which wasn’t even in the TIP database!).  Good users, like Steve, take responsibility for reframing it practically, but I’m not confident that even a majority of ADDIE use is so enabled.  Consequently, I worry that ADDIE is more detrimental than good.  It ensures the minimum, but it essentially prevents inspiration.

I’m willing to be wrong, but I’ve been looking at the debate on both sides for a long time.  While I know that PowerPoint doesn’t kill people, people kill people, and the same is true of ADDIE, the continued reliance on it is problematic.  We probably need a replacement, one that starts with a broader analysis, and then provides guidance across job aid development, course development and more, that has at core iterative and situated design, informed by the recognition of the emotional nature of human use.  Anyone have one to hand?  Thoughts on the above?

22 March 2010

Learning Tools

Clark @ 8:49 AM

Owing to sins in my past, I not only am speaking on mobile learning at the eLearning Guild’s Learning Solutions conference e-Learning Foundations Intensive session, but also introduced the tools section.  The tools will be covered by smart folks like Patti Shank, Harry Mellon, Steve Foreman, and Karen Hyder, but I was supposed to set the context.

Now, I talked about a number of things, including vendors, total cost of ownership, tradeoffs, and the development process, but I also included the following diagram attempting to capture the layers of systems that support tools, and both formal and informal. In some ways the distinctions I make are arbitrary (not to say abstract :), but still, I intended this to be a useful characterization of the space:

The point here is that on top of the hardware and systems are applications. There are assets (with media tools) you create that can (and should) be managed, and then they’re aggregated into content whether courses or resources, that are accessible through synchronous or asynchronous courses or games, portals or feeds, and managed whether through an LMS or a Social Networking System.

The graphic was hard to see on the screen (mea culpa), so I’ve reproduced it here.  Does this make sense?

20 March 2010

The GPS and EPSS

Clark @ 11:07 AM

It’s not unknown for me to enter my name into a drawing for something, if I don’t mind what they’re doing with it.  It’s almost unknown, however, for me to actually win, but that’s actually the case a month or so ago when I put a comment on a blog prior to the MacWorld show, and won a copy of Navigon turn-by-turn navigation software for my iPhone.  I’d thought a dedicated one might be better, though I’d have to carry two devices, but if I moved from an iPhone to Droid or Pre I’d suffer. But for free…

When I used to travel more (and that’s starting again), I’ve usually managed to get by with Google Maps: put in my desired location (so glad they finally put copy/paste in, such a no-brainer rather than have to write it elsewhere and type it on, or remember, usually imperfectly).  In general, maps are a great cognitive augment, a tool we’ve developed to be very useful.  And I’m pretty good with directions (thankfully), so when a trip went awry it wasn’t too bad.  (Though upper New Jersey…well, it can get scary.)  Still, I’d been thinking seriously about getting a GPS, and then I won one!

And I’m happy to report that Navigon is pretty darn cool.  At first the audio was too faint, but then I found out that upping the iPod volume (?) worked.  (And then it didn’t the last time, at all, with no explanation I can find.  Wish it used the darn volume buttons. We’ll see next time. ) However, it does a fabulous job of displaying where you are, what’s coming up, and recalculating if you’ve made a mistake.  It’s a battery hog, keeping the device on all the time, but that’s why we have charging holders (which I’d already acquired for long trips and music).  It also takes up memory, keeping the maps onboard the device (handy if you’re in an area with bad network coverage), but that’s not a problem for me.

However, my point here is not to extol the virtues of a GPS, but instead to use them as a model for some optimum performance support, as an EPSS (Electronic Performance Support System).  There’s a problem with maps in a real-time performance situation. This goes back to my contention that the major role of mlearning is accessorizing our brain.  Memorizing a map of a strange place is not something our brains do well.  We can point to the right address, and in familiar places choose between good roads, but the cognitive overhead is too high for a path of many turns in unfamiliar territories.  To augment the challenge, the task is ‘real time’, in that you’re driving and have to make decisions within a limited window of recognition.  Also, your attention has to be largely outside the vehicle, directed towards the environment. And to cap it all of, the conditions can be dark, and visibility obscured by inclement weather.  All told, navigation can be challenging.

While the optimal solution is a map-equipped partner sitting ‘shot-gun’, a GPS has been designed to be the next best thing (and in some ways superior).  It has the maps, knows the goal, and often more about certain peculiarities of the environment than a map-equipped but similarly novice partner.  A GPS also typically does not get it’s attention distracted when it should be navigating.  It can provide voice assistance while you’re driving, so you don’t need to look at the device when your attention needs to be on the road, but at safe moments it can display useful guidance about lanes to be in (and avoid) visually, without requiring much screen real estate.

And that’s a powerful model to generalize from: what is the task, what are our strengths and limitations, and what is the right distribution of task between device and individual?  What information can a device glean from the immediate and networked environment, from the user, and then provide the user, either onboard or networked?  How can it adapt to a changing state, and continue to guide performance?

Many years ago, Don Norman talked about how you could sit in pretty much any car and know how to drive it, since the interface had time to evolve to a standard.  The GPS has similarly evolved in capabilities to a useful standard.  However, the more we know about how our brains work, the more we can predetermine what sort of support is likely to be useful.  Which isn’t to say that we still won’t need to trial and refine, and use good principles of design across the board, interface, information architecture, minimalism, and more.  We can, and should, be thinking about meeting organizational performance, not just learning needs.  Memorizing maps isn’t necessarily going to be as useful as having a map, and knowing how to read it.  What is the right breakdown between human and tool in your world, for the individuals you want to perform to their best?  What’s their EPSS?

And on a personal note, it’s nice to have the mobile learning manuscript draft put to bed, and be able to get back into blogging and more. A touch of the flu has delayed my ability to think again, but now I’m ready to go.  And off I go to the Learning Solutions conference in Orlando, to talk mobile, deeper learning, and more.  The conference will both interfere with blogging and provide fodder as well.  If you’re there, please do say hello.

5 March 2010

Some accumulated thoughts…

Clark @ 5:35 PM

I have had my head down cranking out the manuscript for my mobile learning book. The deadline for the first draft is breathing down my neck, and I’ve been quite busy with some client work as well.  The proverbial one-armed paper hanger comes to mind.

However, that does not mean my mind has been idle.  Far from, actually.  It’s just not been possible  to find the time to do the thoughts justice.  I’m not really going to here, either, but I do want to toss out some recent thoughts and see what resonates with you, so these are mini-blogs (not microblogging):

A level above

I have long argued that we don’t use mental models enough in our learning, and also that we focus too much on knowledge and not enough on skills.  As I  think about developing learning, I want to equip learners to be able to regenerate the approach they should be using if they forget some part of it, and can if they have been given a conceptual model as relationships that guide the application to a problem.

I realize I want to go further, however.  Given the rate of change of things these days, and the need to empower learners to go beyond just what is presented (moving from training to education, in a sense), I think we need to go further to facilitate the transition from ‘dependent’ learning to independent and interdependent learning, as my colleague Harold Jarche so nicely puts it.

To do that, I think we need to take our presentation of the model a little bit further.  I think we need to look at, as a goal, having presented the learning in such a way that our learners understand the concept not only to regenerate, but maintain, extend, and self-improve.  Yes, it is some extra work, but I think that is going to be critical. It will not only be the role of the university (despite Father Guido), but also the workplace. It’s not quite clear what that means practically, but I definitely want to put this stake into the ground to start thinking about it.  What are your thoughts?

More on the iPad and the Publishing marketplace

I’ve already posted on the iPad, but I want to go on a little longer.  First, the good news: OmniGroup has announced that they’ll be porting OmniGraffle (and their other apps) to the iPad.  Yay!  I *really* like their diagramming tool (where do you think I come up with all those graphics?).

On the other hand, I had lunch the other day with Joe Miller, who is the VP of Tech for Linden Labs.  He recently was talking about the iPad and really sees it as a game changer in ways that are subtle and insightful.  As we talked, he really feels that the whole Flash thing is a big mistake: that one of the things you would use the iPad for is surfing the web, and that more than 75% of the web runs Flash.  It does seem like a relatively small thing to let hang up a major play.

Further, as I said earlier, I think interactivity is the  major opportunity for publishers to go beyond the textbook on eReaders, and the iPad could lead the way.  But right now, Flash is the lingua franca of interactivity on the web, and without it, there’s not an obvious fallback that won’t require rewriting across platforms instead of write-once, run anywhere.

Joe did point me to an interesting new eReader proposal, by Ray Kurzweil of all people.  Oddly, it’s Windows-only, so not quite sure the relevance to the Mac (tho’ you’d think they’d port it over with alacrity), but a free, more powerful eReader platform could have a big impact.

Lots of more interesting things on the way, after I get this draft off to the publisher and get back into the regular blogging swing. ‘Til then, take care,  and keep up the dialog!

Powered by WordPress