Clark Quinn's Learnings about Learning
(The Official Quinnovation blog)

24 August 2017

Extending Engagement

Clark @ 8:09 AM

My post on why ‘engagement’ should be added to effective and efficient led to some discussion on LinkedIn. In particular, some questions were asked that I thought I should reflect on.  So here are my responses to the issue of how to ‘monetize’ engagement, and how it relates to the effectiveness of learning.

So the first issue was how to justify the extra investment engagement would entail. It was an assumption that it would take extra investment, but I believe it will. Here’s why. To make a learning experience engaging, you need some additional things: knowing why this is of interest and relevance to practitioners, and putting that into the introduction, examples, and practice.  With practice, that’s going to come with only a marginal overhead. More importantly, that is part of also making it more effective. There is some additional information needed, and more careful design, and that certainly is more than most of what’s being done now. (Even if it should be.)

So why would you put in this extra effort?  What are the benefits? As the article suggested, the payoffs are several:

  • First, learners know more intrinsically why they should pay attention. This means they’ll pay more attention, and the learning will be more effective. And that’s valuable, because it should increase the outcomes of the learning.
  • Second, the practice is distributed across more intriguing contexts. This means that the practice will have higher motivation.  When they’re performing, they’re motivated because it matters. If we have more motivation in the learning practice, it’s closer to the performance context, so we’re making the transfer gap smaller. Again, this will make the learning more effective.
  • Third, that if you unpack the meaningfulness of the examples, you’ll make the underlying thinking easier to assimilate. The examples are comprehended better, and that leads to more effectiveness.

If learning’s a probabilistic game (and it is), and you increase the likelihood of it sticking, you’re increasing the return on your investment. If the margin to do it right is less than the value of the improvement in the learning, that’s a business case. And I’ll suggest that these steps are part of making learning effective, period. So it’s really going from a low likelihood of transfer – 20-30% say – to effective learning – maybe 70-80%.  Yes, I’m making these numbers up, but…

This is really all part of going from information dump & knowledge test to elaborated examples and contextualized practice.  So that’s really not about engagement, it’s about effectiveness. And a lot of what’s done under the banner of ‘rapid elearning’ is ineffective.  It may be engaging, but it isn’t leading to new skills.

Which is the other issue: a claim that engagement doesn’t equal better learning. And in general I agree (see: activity doesn’t mean effectiveness in a social media tool). It depends on what you mean by engagement; I don’t mean trivialized scores equalling more activity. I mean fundamental cognitive engagement: ‘hard fun’, not just fun.  Intrinsic relevance. Not marketing flare, but real value add.

Hopefully this helps!  I really want to convince you that you want deep learning design if you care about the outcomes.  (And if you don’t, why are you bothering? ;).  It goes to effectiveness, and requires addressing engagement. I’ll also suggest that while it does affect efficiency, it does so in marginal ways compared to substantial increases in impact.  And that strikes me as the type of step one should be taking. Agreed?


16 August 2017

3 E’s of Learning: why Engagement

Clark @ 8:07 AM

Letter EWhen you’re creating learning experiences, you want to worry about the outcomes, but there’s more to it than that.  I think there are 3 major components for learning as a practical matter, and I lump these under the E’s: Effectiveness, Efficiency, & Engagement. The latter may be more of a stretch, but I’ll make the case .

When you typically talk about learning, you talk about two goals: retention over time, and transfer to all appropriate (and no inappropriate) situations.  That’s learning effectiveness: it’s about ensuring that you achieve the outcomes you need.  To test retention and transfer, you have to measure more than performance at the end of the learning experience. (That is, unless your experience definition naturally includes this feedback as well.) Let alone just asking learners if they thought it was valuable.  You have to see if the learning has persisted later, and is being used as needed.

However, you don’t have unlimited resources to do this, you need to balance your investment in creating the experience with the impact on the individual and/or organization.  That’s efficiency. The investment is rewarded with a multiplier on the cost.  This is just good business.

Let’s be clear: investing without evaluating the impact is an act of faith that isn’t scrutable.  Similarly, achieving the outcome at an inappropriate expense isn’t sustainable.  Ultimately, you need to achieve reasonable changes to behavior under a viable expenditure.

A few of us have noticed problems sufficient to advocate quality in what we do.  While things may be trending upward (fingers crossed), I think there’s still ways to go when we’re still hearing about ‘rapid’ elearning instead of ‘outcomes’.  And I’ve argued that the necessary changes produce a cost differential that is marginal, and yet yields outcomes more than marginal.   There’s an obvious case for effectiveness and efficiency.

But why engagement? Is that necessary? People tout it as desirable. To be fair, most of the time they’re talking about design aesthetics, media embellishment, and even ‘gamification‘ instead of intrinsic engagement.  And I will maintain that there’s a lot more possible. There’s an open question, however: is it worth it?

My answer is yes. Tapping into intrinsic interest has several upsides that are worth the effort.  The good news is that you likely don’t need to achieve a situation where people are willing to pay money to attend your learning. Instead, you have the resources on hand to make this happen.

So, if you make your learning – and here in particular I mean your introductions, examples, and practice – engaging, you’re addressing motivation, anxiety, and potentially optimizing the learning experience.

  • If your introduction helps learners connect to their own desires to be an agent of good, you’re increasing the likelihood that they’ll persist and that the learning will ‘stick’.
  • If your examples are stories that illustrate situations the learner recognizes as important, and unpack the thinking that led to success, you’re increasing their comprehension and their knowledge.
  • Most importantly, if your practice tasks are situated in contexts that are meaningful to learners both because they’re real and important, you’ll be developing their skills in ways closest to how they’ll perform.  And if the challenge in the progression of tasks is right, you’ll also accelerate them at the optimal speed (and increase engagement).

Engagement is a fine-tuning, and learner’s opinions on the experience aren’t the most important thing.  Instead, the improvement in learning outcomes is the rationale.  It takes some understanding and practice to get systematically good at doing this. Further, you can make learning engaging, it is an acquired capability.

So, is your learning engaging intrinsic interest, and making the learning persist? It’s an approach that affects effectiveness in a big way and efficiency in a small way. And that’s the way you want to go, right? Engage!

9 August 2017

Simulations versus games

Clark @ 8:04 AM

At the recent Realities 360 conference, I saw some confusion about the difference between a simulation and a game. And while I made some important distinctions in my book on the topic, I realize that it’s possible that it’s time to revisit them. So here I’m talking about some conceptual discriminations that I think are important.


As I’ve mentioned, simulations are models of the world. They capture certain relationships we believe to be true about the world. (For that matter, they can represent worlds that aren’t real, certainly the case in games.). They don’t (can’t) capture all the world, but a segment we feel it is important to model. We tend to validate these models by testing them to see if they behave like our real world.  You can also think about simulations as being in a ‘state’ (set of values in variables), and move to others by rules.  Frequently, we include some variability in these models, just as is reflected in the real world. Similarly, these simulations can model considerable complexity.

Such simulations are built out of sets of variables that represent the state of the world, and rules that represent the relationships present. There are several ways things change. Some variables can be changed by rules that act on the basis of time (while countdown timer = on, countdown = countdown -1). Variables can also interact (if countdown=0: if 1 g adamantium and 1 g dilithium, Temperature = Temperature +1000, adamantium = adamantium – 1g, dilithium = dilithium – 1g).  Other changes are based upon learner actions (if learner flips the switch, countdown timer = on).

Note that you may already have a simulation. In business, there may already exist a model of particular processes, particularly if they’re proprietary systems.

From a learning point of view, simulations allow motivated and self-effective learners to explore the relationships they need to understand. However, we can’t always assume motivated and self-effective learners. So we need some additional work to turn a simulation into a learning experience.


One effective way to leverage simulations is to choose an initial state (or ‘space of states’, a start point with some variation), and a state (or set) that constitutes ‘win’. We also typically have states that also represent ‘fail’.  We choose those states so that the learner can’t get to ‘win’ without understanding the necessary relationships.   The learner can try and fail until they discover the necessary relationships.  These start and goal states serve as scaffolding for the learning process.  I call these simulations with start and stop states ‘scenarios’.

This is somewhat complicated by the existence of ‘branching scenarios’. There are initial and goal states and learner actions, but they are not represented by variable and rules. The relationships in branching scenarios are implicit in the links instead of explicit in the variables and rules. And they’re easier to build!  Still, they don’t have the variability that typically is possible in a simulation. There’s an inflection point (qualitative, not quantitative) where the complexity of controlling the branches renders it more sensible to model the world as a simulation rather than track all the branches.


The problem here is that too often people will build a simulation and call it a game. I once reviewed a journal submission about a ‘game’ where the authors admitted that players thought it was boring. Sorry, then it’s not a game!  The difference between a simulation and a game is a subjective experience of engagement on the part of the player.

So how do you get from a simulation to a game?  It’s about tuning.  It’s about adjusting the frequency of events, and their consequences, such that the challenge moves to fall into the zone between boring and frustrating. Now, for learning, you can’t change the fundamental relationships you’re modeling, but you can adjust items like how quickly events occur, and the importance of being correct. And it takes testing and refinement. Will Wright, a game designers’ game designer, once proposed that tuning is 9/10’s of the work!  Now that’s for a commercial game, but it gives you and idea.

You can also use gamification, scores to add competition, but, please, only after you first expend the effort to make the game intrinsically interesting. Tap into why they should care about the experience, and bake that it.

Is it worth it to actually expend effort to make the experience engaging?  I believe that the answer is yes. Perhaps not to the level of a game people will pay $60 to play, but some effort to manifest the innate meaningfulness is worth it. Games minimize the time to obtain competency because they optimize the challenge.  You will have sticks as well as carrots, so you don’t need to put in $M budgets, but do tune until your learners have an engaging and effective experience.

So, does this help? What questions do you still have?

8 August 2017

L&D Tuneup

Clark @ 8:00 AM

auto engineIn my youth, owing to my father’s tutelage and my desire for wheels, I learned how to work on cars. While not the master he was, I could rebuild a carburetor, gap points and sparkplugs, as well as adjust the timing. In short, I could do a tuneup on the car.  And I think that’s what Learning & Development (L&D) needs, a tuneup.

Cars have changed, and my mechanic skills are no longer relevant. What used to be done mechanically – adjusting to altitude, adapting through the stages of the engine warming up, and handling acceleration requests – are now done electronically. The air-fuel mixture and the spark advance are under the control of the fuel injection and electronic ignition systems (respectively) now.  With numerous sensors, we can optimize fuel efficiency and performance.

And that’s the thing: L&D is too often still operating in the old, mechanical, model. We have the view of a hierarchical model where a few plan and prepare and train folks to execute. We stick with face-to-face training or maybe elearning, putting everything in the head, when science shows that we often function better from information in the world or even in other people’s heads!  And this old approach no longer works.

As has been noted broadly and frequently, the world is changing faster and the pressure is on organizations to adapt more quickly. With widely disparate paths  pointing in the same direction, it’s easy to see that there’s something fundamental going on. In short, we need to move, as Jon Husband puts it, from hierarchy to wirearchy.  We need agility: experimentation, review, and reflection, iteratively and collectively. And in that move, there’s a central role for L&D.

The move may not be imminent, but it is unavoidable. Even staid and secure organizations are facing the consequences of increasing rates of change and new technology innovations. AI, networks, 3D printing, there are ramifications. Even traditional government agencies are facing change. Yet, this is all about people and learning.

As Harold Jarche tells us, work is learning and learning is the work. That means learning is moving from the classroom to the workplace and on the go. L&D needs a modern workplace learning approach, as Jane Hart lets us know. This new model is one where L&D moves from fount of knowledge to learning facilitator (or advisor, as she terms it).  People need to develop those communication and collaboration, but it won’t come from classes, but from coaching and more.

And, to return to the metaphor, I view this as an L&D tuneup. It’s not about throwing out what you’re doing (unless that’s the fastest path ;), but instead augmenting it. Shifts don’t happen overnight, but instead it means taking on some internal changes, and then working that outwards with stakeholders, reengineering the organizational relationships. It’s a journey, not an event. But like with a tuneup, it’s about figuring out what your new model should be, and then adjusting until you achieve it. It’s over a more extended period of time, but it’s still a tuning operation. You have to work through the stages to a new revolutionary way of working. So, are you ready for a tuneup?

1 August 2017

Realities 360 Reflections

Clark @ 8:08 AM

So, one of the two things I did last week was attend the eLearning Guild‘s Realities 36o conference.  Ostensibly about Augmented Reality (AR) and Virtual Reality (VR), it ended up being much more about VR. Which isn’t a bad thing, it’s probably as much a comment on the state of the industry as anything.  However, there were some interesting learnings for me, and I thought I’d share them.

First, I had a very strong visceral exposure to VR. While I’ve played with Cardboard on the iPhone (you can find a collection of resources for Cardboard here), it’s not quite the same as a full VR experience.  The conference provided a chance to try out apps for the HTC Vive, Sony Playstation VR, and the Oculus.  On the Vive, I tried a game where you shot arrows at attackers.  It was quite fun, but mostly developed some motor skills. On the Oculus, I flew an XWing fighter through an asteroid field and escorted a ship and shoot enemy Tie-fighters.  Again, fun, but mostly about training my motor skills in this environment.

It was the one I think on the Vive that gave me an experience.  In it, you’re floating around the International Space Station. And it was very cool to see the station and experience the immersion of 3D, but it was very uncomfortable.  Partly because I was trying to fly around (instead of using handholds), my viewpoint would fly through the bulkhead doors. However, the positioning meant that it gave the visual clues that my chest was going through the metal edge.  This was extremely disturbing to me!  As I couldn’t control it well, I was doing this continually, and I didn’t like it. Partly it was the control, but it was also the total immersion. And that was impressive!

There are empirical results that demonstrate better learning outcomes for VR, and certainly  I can see that particularly, for tasks inherently 3D. There’s also another key result, as was highlighted in the first keynote: that VR is an ’empathy’ machine. There have been uses for things like understanding the world according to a schizophrenic, and a credit card call center helping employees understand the lives of card-users.

On principle, such environs should support near transfer when designed to closely mimic the actual performance environment. (Think: flight or medicine simulators.)  And the tools are getting better. There’s an app that allows you to take photos of a place to put into Cardboard, and game engines (Unity or Unreal or both) will now let you import AutoCAD models.  There was also a special camera that could sense the distances in a space and automatically generate a model of it.  The point being that it’s getting easier and easier to generate VR environments.

That, I think, is what’s holding AR back.  You can fairly easily use it for marker or location based information, but actually annotating the world visually is still challenging.  I still think AR is of more interest, (maybe just to me), because I see it eventually creating the possibility to see the causes and factors behind the world, and allow us to understand it better.  I could argue that VR is just extending sims from flat screen to surround, but then I think about the space station, and…I’m still pondering that. Is it revolutionary or just evolutionary?

One session talked about trying to help folks figure out when VR and AR made sense, and this intrigued me. It reminded me that I had tried to characterize the affordances of virtual worlds, and I reckon it’s time to take a stab at doing this for VR and AR.  I believed then that I was able to predict when virtual worlds would continue to find value, and I think results have borne that out.  So, the intent is to try to get on top of when VR and AR make sense.  Stay tuned!

27 July 2017

Barry Downes #Realities360 Keynote Mindmap

Clark @ 9:59 AM

Barry Downes talked about the future of the VR market with an interesting exploration of the Immersive platform. Taking us through the Apollo 11 product, he showed what went into it and the emotional impact. He showed a video that talked (somewhat simplistically) about how VR environments could be used for learning. (There is great potential, but it’s not about content.). He finished with an interesting quote about how VR would be able to incorporate any further media. A second part of the quote said: “Kids will think it’s funny [we] used to stare at glowing rectangles hoping to suspend disbelief.”

VR Keynote

26 July 2017

Maxwell Planck #Realities360 Keynote Mindmap

Clark @ 9:59 AM

Maxwell Planck opened the eLearning Guild’s Realities 360 conference with a thoughtful and thought-provoking talk on VR. Reflecting on his experience in the industry, he described the transition from story telling to where he thinks we should go: social adventure. (I want to call it “adventure together”. :). A nice start to the event.

Maxwell Planck Keynote Mindmap

20 July 2017

Augmented Reality Lives!

Clark @ 8:07 AM

Visually Augmented RealityAugmented Reality (AR) is on the upswing, and I think this is a good thing. I think AR makes sense, and it’s nice to see both solid tool support and real use cases emerging.  Here’s the news, but first, a brief overview of why I like AR.

As I’ve noted before, our brains are powerful, but flawed.  As with any architecture, any one choice will end up with tradeoffs. And we’ve traded off detail for pattern-matching.  And, technology is the opposite: it’s hard to get technology to do pattern matching, but it’s really good at rote. Together, they’re even more powerful. The goal is to most appropriately augment our intellect with technology to create a symbiosis where the whole is greater than the sum of the parts.

Which is why I like AR: it’s about annotating the world with information, which augments it to our benefit.  It’s contextual, that is, doing things because of when and where we are.  AR augments sensorily, either auditory or visual (or kinesthetic, e.g. vibration).  Auditory and kinesthetic annotation is relatively easy; devices generate sounds or vibrations (think GPS: “turn left here”).  Non-coordinated visual information, information that’s not overlaid visually, is presented as either graphics or text (think Yelp: maps and distances to nearby options).  Tools already exist to do this, e.g. ARIS.  However, arguably the most compelling and interesting is the aligned visuals.

Google Glass was a really interesting experiment, and it’s back.  The devices – glasses with camera and projector that can present information on the glass – were available, but didn’t do much because of where you were looking. There were generic heads-up displays and camera, but little alignment between what was seen and what was consequently presented to the user with additional information.  That’s changed. Google Glass has a new Enterprise Edition, and it’s being used to meet real needs and generate real outcomes. Glasses are supporting accurate placement in manufacturing situations requiring careful placement.  The necessary components and steps are being highlighted on screen, and reducing errors and speeding up outcomes.

And Apple has released it’s Augmented Reality software toolkit, ARKit, with features to make AR easy.  One interesting aspect is built-in machine learning, which could make aligning with objects in the world easy!  Incompatible platforms and standards impede progress, but with Google and Apple creating tools for each of their platforms, development can be accelerated. (I hope to find out more at the eLearning Guild’s Realities 360 conference.)

While I think Virtual Reality (VR) has an important role to play for deep learning, I think contextual support can be a great support for extending learning (particularly personalization), as well as performance support.  That’s why I’m excited about AR. My vision has been that we’ll have a personal coaching system that will know where and when we are and what our goals are, and be able to facilitate our learning and success. Tools like these will make it easier than ever.

11 July 2017

A Bad Tart

Clark @ 8:00 AM

Good learning requires a basis for intrinsic interest. The topic should be of interest to the learner, a priori or after the introduction. If the learner doesn’t ‘get’ why this learning is relevant to them, it doesn’t stick as well. And this isn’t what gamification does. So tarting up content is counter-productive. It’s a bad (s)tart!

Ok, to be clear, there’re two types of gamification. The first, important, and relevant type of gamification is using game design techniques to embed learning topic into meaningful series of decisions, where the context and actions taken affect the outcomes in important ways, and the challenge is appropriate.  However, that’s not the one that’s getting all the hype.

Instead, the hype is around PBL (which, sadly isn’t Problem-Based Learning but instead is Points, Badges, & Leaderboards).  If we wrap this stuff around our learning, we’ll make it more engaging.  And, at least initially, we’ll see that. At least in enthusiasm. But how about retention and transfer?  And will there be a drop-off when the novelty wears off?

Yes, we can tart up drill-and-kill, and should, if that’s what’s called for (e.g. accurate retention of information). But that’s not what works for skills. And the times it’s actually relevant are scarce. For skills, we want appropriate retrieval.  And that means something else.

Retention and transfer of new skills requires contextualized retrieval and application to decisions that learners need to be able to make. And that’s scenarios (or, at least, mini-scenarios).  We need to put learners into situations requiring applying the knowledge to make decisions. Then the consequences play out.

If you’re putting your energy into finding gratuitous themes to wrap around knowledge recitation instead of making intrinsically meaningful contexts for knowledge application, you’re wasting time and money.  You’re not going to develop skills.

I actually don’t mind if you want to tart up after you’ve done the work of making the skill practice meaningful. But only after!  If you’re skipping the important practice design, you’re letting down your learners. As well as the organization.  And typically we don’t need to spend unnecessary time.

Please, for your learners’ sake, find out about both sorts of gamification, distinguish between them, and then use them appropriately. PBL is ok when rote knowledge has to be drilled, or after you’ve done good practice design.

6 July 2017

Writing For Learning: Patti Shank book

Clark @ 8:02 AM

While I ordinarily refuse (on principle, otherwise I’d get swamped and become a PR hack; and I never promise a good review), I acquiesced to Patti Shank’s offer of a copy of her new book Write and Organize for Deeper Learning.  I’ve been a fan of her crusade for science in learning (along with others like Will Thalheimer, Julie Dirksen, and Michael Allen, to name a few co-conspirators on the Serious eLearning Manifesto), and I can say I’m glad she offered and delivered.  This is a contribution to the field, with a focus on writing.

The first of a potential series on evidence-based processes in learning design, this one is focused on content: writing and organization. In four overall strategies with 28 total tactics, she takes you through what you should do and why. Practicing what she preaches, and using the book itself recursively as an example, she uses simple words and trimmed down prose to focus on what you need to know, and guidance to generalize it. She also has practice exercises to help make the  material stick.

Which isn’t to say there aren’t quibbles: for one, there’s no index!  I wanted to look for ‘misconceptions’ (an acid test in my mind), but it’s in there. Still, there’s less on cognitive models to guide performance than I would like. And not enough on misconceptions, But there’re lots of good tips that I wouldn’t have thought of including, and are valuable.

These are small asides. Some of it’s generic (writing for the web, for instance, similarly argues for whitespace) and some specific to learning, but it’s all good advice, and insufficiently seen.  What’s there will certainly improve the quality of your learning design.  If you write prose for elearning, you definitely should read, and heed, this book.   I note, by the way, that my readability index for this blog always falls too high according to her standards ;).

She has a list of other potential books, and I can hope that she will at least deliver the one on designing practice and feedback (what I think is the biggest opportunity to improve elearning: what people do), but also examples, job aids, evaluating, objectives, and more.  I hope this series continues to develop, based upon the initial delivery here.



Next Page »

Powered by WordPress