Learnlets

Secondary

Clark Quinn’s Learnings about Learning

Search Results for: align

Making Multiple Choice work

by Leave a Comment

For sins in my past, I’ve been thinking about assessments a bit lately. And one of the biggest problems comes from trying to find solutions that are meaningful yet easy to implement. You can ask learners  to  develop meaningful artifacts, but getting them assessed at scale is problematic. Mostly, auto-marked stuff is used to do trivial knowledge checks. Can we do better.

To be fair, there are more and more approaches (largely machine-learning powered), that can do a good job of assessing complex artifacts, e.g. writing. If you can create good examples, they can do a decent job of learning to evaluate how well a learner has approximated it. However, those tools aren’t ubiquitous. What is are the typical variations on multiple choice: drag and drop, image clicks, etc. The question is, can we use these to do good things?

I want to say yes. But you have to be thinking in a different way than typical. You can’t be thinking about testing knowledge recognition. That’s not as useful a task as knowledge retrieval. You don’t want learners to just have to discriminate a term, you want them to  use the knowledge to do something. How do we do that?

In  Engaging Learning, amongst other things I talked about ‘mini-scenarios’. These include a story setting and a required decision, but they’re singular, e.g. they don’t get tied to subsequent decisions. And this is just a better form of multiple choice!

So, for example, instead of asking whether an examination requires an initial screening, you might put the learner in the role of someone performing an examination, and have alternative choices of action like beginning the examination, conducting an initial screening, or reviewing case history. The point is that the learner is making choices  like the ones they’ll be making in real practice!

Note that the alternatives aren’t random; but instead represent ways in which learners reliably go wrong. You want to trap those mistakes in the learning situation, and address them  before they matter!  Thus, you’re not recognizing whether it’s right or not, you’re using that information to discriminate between actions that you’d take.  It may be a slight revision, but it’s important.

Further, you have the consequences of the choice play out: “your examination results were skewed because…and this caused X”.  Then you can give the principled feedback (based upon the model).

There are, also, the known obvious things to do. That is, don’t have any ‘none of the above’ or ‘all of the above’. Don’t make the alternatives obviously wrong. And, as Donald Clark summarizes, have two alternatives, not three. But the important thing, to me, is to have different choices based upon using the information to make decisions, not just recognizing the information amongst distractors. And capturing misconceptions.

These can be linked into ‘linear’ scenarios (where the consequences make everything right so you can continue in a narratively coherent progression) or branching, where decisions take you to different new decisions dependent on your choice.  Linear and branching scenarios are powerful learning. They’re just not always necessary or feasible.

And I certainly would agree that we’d like to do better: link decisions and complex work products together into series of narratively contextualized settings, combining the important types of decisions that naturally occur (ala Schank’s Goal Based Scenarios and Story-Centered Curriculum and other similar approaches).  And we’re getting tools that make this possible. But that requires some new thinking. This is an interim step that, if you get your mind around it, sets you up to start wanting more.

Note that the thinking here also covers a variety of interaction possibilities, again drag’n’drop, image links, etc. It’s a shift in thinking, but a valuable one. I encourage you to get your mind around it. Better practice, after all, is better learning.

Intellectricity

by Leave a Comment

Many years ago, I met a guy who worked for Apple. They were allowed to have their own job titles, and his was “Intellectrician”. I thought that was a very nice turn of phrase. And, as I just ordered new business cards, I put  Intellectricity as the tagline instead of “Learning & Performance Strategy” or other permutations with Technology and such in the title.  Why?

The goal, of course, is to have a phrase that folks will read it and go “what’s that about?”, as some of my colleagues helped me remember. If you can spark a conversation, you have a chance to do a little evangelism/education. (And maybe some business interest?)  Also, I think it actually captures what I believe and like to do pretty well.

You’ve likely heard or read me harp many times on how companies aren’t well aligned with how we think, work, and learn. The cognitive violations are many, from how we design our learning, to design our workplaces, policies, tool use, and culture. If we redesigned what we’re doing, creating strategies to get better practices in place, we’d be unleashing the organizational intellect!  Hence, ‘intellectricity’.

And this is pretty much what I’m on about, in several ways:

  • knowing what formal learning  really  looks like, and designing our design processes accordingly
  • recognizing what facilitates informal learning in the short term (the ‘solve this’ type of problem-solving’)
  • facilitating long-term informal learning by practices and tools suites
  • fostering a culture where innovation thrives

This is a partial list that goes fractal really quickly with practices and principles around each area. The point is that these elements are key to organizational ‘thrival’.  Overall, they’re about optimizing the intellectual activity of the organization, learning quickly to be agile.

We’ll see if this tactic works to generate conversations and then new thinking. As Jay Cross used to say “conversations are the stem cells of learning”.  Practicing what we preach.  So here’s to Intellectricity: more conversations and more learning.

Competencies and Innovation?

by Leave a Comment

This may seem like an odd pairing, so bear with me.  I believe that we want to find ways to support organizations moving in the direction of innovation and learning cultures. Of course, I’ve been on a pretty continuous campaign for this, but I’m wondering what other levers we have. And, oddly, I think competencies may be one. Let me make the case for competencies and innovation.

So I’ve gotten involved in standards and competency work. Don’t ask me why, as I have no better answer than a) they asked, and b) the big ‘sucker’ tattoo on my forehead.  Of course, as I’ve said before, the folks that do this stuff (besides me, obviously) are really contributing to the benefit of our org. Maybe I felt I had to walk the talk?

In the course of the one that was just launched, we identified a number of competencies across the suite of L&D activities. This included (in addition the more traditional activities) looking at how to foster innovation. This means understanding culture and the change processes to get there, as well as knowing how to run meetings that get the best outputs. It’s about being prepared for both types of innovation, fast (solve ‘this’ problem) and slow (the steady percolation of ideas).

Thus, the necessary skills are identified as a component of a full suite of L&D capabilities. And the hope, of course, is that people will begin to recognize that there are parts of L&D they’re not addressing, and move to take on this opportunity. I hope that it’s becoming obvious that the ability to facilitate innovation is an organizational imperative, and that there’s a strong argument for L&D to be key. This is on principle, and pragmatically, it’s a no-brainer for L&D to find a way to become central to org success, not peripheral.

However, leaving that to chance would be, well, just silly. What can we do?  Well, two things, I think: one is to help raise awareness, the other is to provide support. A suite of skills aligned to this area is a ‘good thing’ if it known and used. Working on the know has been an ongoing thing (*cough*), but how can we support it?

Again, two things, I think. One are examples where people have put in place programs where they’ve oriented themselves in this direction and documented benefits. The other is to provide scaffolding; support materials that help folks implement these competencies. And I believe that’s coming.

“Systematic creativity is  not an oxymoron” (I may need to make a quip post about that). And this is an example. Think of brainstorming, for example. It can be useful, or  not. When done right, the outcomes are much better. And similarly in lots of ways, the nuances matter. If we define, through competencies, what suites of knowledge matter, we bring awareness to the possible outcomes. And the opportunity to improve them.

It may be an indirect path, to be sure, but it’s a steady, and real one. In fact, to say “we want to innovate, but how” and have a suite of specific sets of knowledge on tap to point people to, is pretty much next to the fastest path.  Showing people the benefits and the path to obtain them is key. It’s even self-referential: let’s innovate on making innovation systematically embedded in organizations! ;)  So, keep on experimenting!

Why Engaging Learning?

by Leave a Comment

Book coverSomeone asked me what I would say about my first book, Engaging Learning. And, coincidentally, my client just gave some copies to their client as part of our engagement, so I guess there’s still value in it!  And while I recognize it’s now about 13 years old, I really do believe it has relevance. Since they asked…

I saw the connections between computers and learning as an undergraduate, and designed my own major. My first job out of college was designing and programming educational computer games. Long story short: I went back for a Ph.D. in what was effectively ‘applied cognitive science’, but games continued to play a role in my career. And I reflected on it, and ultimately what started as a research agenda manifested as a model for explaining why games work and how to do it. And then when I started consulting, Pfeiffer asked me to write the book.

To be clear, I believe engagement matters.  We learn better when our hearts and our minds are engaged. (That’s the intent of the double meaning of the title, after all.)  Learning sticks when we’re motivated and in a ‘safe’ learning situation.  Learning can, and should, be ‘hard fun’.  However,  if we can’t do it reliably and repeatedly, it’s just a dream. I believe that if we systematically apply the principles in the book, we can do it (systematic creativity is  not an oxymoron ;).

One of the concerns was that things were changing fast even then (Flash was still very much in play, for example ;).  How to write something that wouldn’t be outdated even before it came out?  So I tied it to cognitive principles, as our brains aren’t changing that fast.  Thus, I think the principles in it still hold.  I’ve continued to check and haven’t found anything that undermines the original alignment that underpins designing engaging experiences.

And the book was designed for use. While the first three chapters set the stage, the middle three dig into details. There you’ll find the core framework, examples, and a design process. The design process was focused mostly on adding to what you already do, so as not to be redundant. The final three chapters wrap up pragmatics and future directions.

While ostensibly (and realistically) about designing games, it was really about engagement. For instance, the principles included were applied backwards to branching scenarios, and what I called linear and mini-scenarios. The latter just being better written multiple choice questions!

The book couldn’t cover everything, and I’ve expanded on my thinking since then, but I believe the core is still there: the alignment and the design process in particular. There have been newer books since then by others (I haven’t stayed tied to just games, my mind wanders more broadly ;) and by me, but as with my other books I think the focus on the cognitive principles gives lasting guidance that still seems to be relevant. At a recent event, someone told me that while I viewed mobile as a known, for others it wasn’t. I reckon that may be true for games and engagement as well. If we’re making progress, I’m pleased. So, please, start engaging learning by making engaging learning!

PS, I wrote a Litmos blog post about why engagement matters, as a prelude to a session I’ll be giving at their Litmos Live  online event (Nov 7-8) where I talk about how to do it.

 

Constraints on activities

by 2 Comments

When we design learning activities (per the activity-based learning model), ideally we’re looking to create an integration of a number of constraints around that assignment. I was looking to enumerate them, and (of course) I tried diagramming it.  Thought I’d share the first draft, and I welcome feedback!

Multiple constraints on assignmentsThe goal is an assignment that includes the right type of processing. This must align with what they need to be able to do after the learning experience. Whether at work or in a subsequent class. Of course, that’s factored into the objective for this learning activity (which is part of an overall sequence of learning).

Another constraint is making sure the setting is a context that helps establish the breadth of transfer. The choice should be sufficiently different from contexts seen in examples and other practices to facilitate abstracting the essential elements. And, of course, it’s ideally in the form of a story that the learner’s actions are contributing to (read: resolve). The right level of exaggeration could play an (unrepresented) role in that story.

We also need the challenge in the activity to be in the right range of difficulty for the learner. This is the integration of flow and learning to create meaningful engagement.  And we want to include ways in which learners typically go wrong (read: misconceptions). Learners need to be able to make the mistakes here so we’re trapping and addressing them in the learning situation, not when it could matter.

Finally, we want to make sure there’s enough variation across tasks. While some similarities benefit for both consistency and addressing the objective, variety can maintain interest. We need to strike that balance. Similarly, look at the overall workload: how much are we expecting, and is that appropriate given the other constraints outside this learning goal.

I think you can see that successfully integrating these is non-trivial, and I haven’t even gotten into how to evaluate this, particularly to make it a part of an overall assessment. Yet, we know that multiple constraints help make the design easier (at least until you constrain yourself to an empty solution set ;).  This is probably still a mix of art and science, but by being explicit you’re less likely to miss an element.

We want to align activities with the desired outcome, in the full context.  So, what am I missing?  Does this make sense?

 

Labels, models, and drives

by Leave a Comment

In my post last week on engagement, I presented the alignment model from my  Engaging Learning  book on designing learning experiences. And as I thought about the post, I pondered several related things about labels, models, and drives. I thought I’d wrestle with them ‘out loud’ here, and troll (in the old sense) to see what you think.

Some folks have branded a model and lived on that for their career. And, in a number of cases, that’s not bad: they’re useful models and their applicability hasn’t diminished. And while, for instance, I think that alignment model is as useful as most models I’ve seen, I didn’t see any reason to tie my legacy to it, because the principles I like to comprehend and then apply to create solutions aren’t limited to just engagement. Though I wonder if people would find it easier to put the model in practice if it had a label.  The Quinn Engagement model or somesuch?

I’ve also created models around mobile, and about performance ecosystems, and more. I can’t say that they’re all original (e.g. the 4Cs of mobile), though I think they have utility. And some have labels (again, the 4Cs, Least Assistance Principle…) Then the misconceptions book is very useful, but the coverage there isn’t really mine, either. It’s just a useful compendium. I expect to keep creating models. But it’d led to another thought…

I’ve seen people driven to build companies. They just keep doing it, even if they’ve built one and sold it, they’re always on it; they’re serial entrepreneurs. I, for instance, have no desire to do that. There are elements to that that aren’t me.    Other folks are driven to do research: they have a knack for designing experiments that tease out the questions that drive them to find answers. And I’ve been good at that, but it’s not what makes my heart beat faster. I do  like action research, which is about doing with theory, and reflecting back. (I also like helping others become able to do this.)

What I’m about is understanding and applying cognitive science (in the broad sense) to help people do important things in ways that are enabled by new technologies.  Models that explain disparate domains are a hobby. I like finding ways to apply them to solve new problems in ways that are insightful but also pragmatic.   If I create models along the way (and I do), that’s a bonus. Maybe I should try to create a model about applying models or somesuch. But really, I like what I do.

The question I had though, is whether anyone’s categorized ‘drives’.  Some folks are clearly driven by money, some by physical challenges. Is there a characterization?  Not that there needs to be, but the above chain of thought led me to be curious. Is there a typology of drives? And, of course, I’m skeptical if there is one (or more), owing to the problems with, for instance, personality types and learning styles :D. Still, welcome any pointers.

Engagement

by 1 Comment

In a meeting today, I was asked “how do you define engagement”, and I found it an intriguing question. I don’t know that I have a definition so much as steps to enhance it. Still, it made me think.

What engagement is not, let’s be clear, is tarting content up. It’s not just flashy visuals, stereotypes, and cute prose.  Those things add aesthetics (or, done poorly, undermine same), but that’s not where to go.

Flow stateInstead, I’m looking for an experience that has certain characteristics. One way of looking at it is through the ‘flow’ phenomenon, with cognitive immersion at a level that finds the sweet spot between frustration and boring.  Similarly, for learning, it’s the Zone of Proximal Development, between what you can do with one hand tied behind your back, and what you can’t do no matter how much support you get.  And it’s both.

You there by exploiting the alignment between the elements of practice and engaging experiences. So just as the above diagram can represent either Czikszentmihalyi or Vygotsky, there’s the alignment I highlighted in Engaging Learning  between the elements in greater elaboration. It’s goal, context, challenge, meaningfulness, and more all aligned to create that subjective feeling. And in case you say “you’re extending engagement to learning”, I will note that Koster, in his book A Theory of Fun, explicitly tied what makes games work  is that it’s about learning. So, yeah, that’s the type of engagement I’m interested in, regardless.

One of the simple ways I like to characterize it (and it’s not original with me), is ‘hard fun’.  I think, if nothing else, that’s a great heuristic. It may be like the famous quote about pornography: “you know it when you see it”. Or maybe you can coin a concise definition. And you can attempt to quantify it through objective criteria like galvanic skin response or adrenalin levels. However, I’m perfectly happy to use subjective criteria. If people say they found it challenging but fun, I’m happy. If they say it’s the best way they can see to learn it, my job is done.

I don’t really yet have a good way to define engagement in a concise specification. Do you have a definition of engagement you like?  I’d welcome hearing it!

 

 

ONE level of exaggeration

by 5 Comments

I’ve argued before that we should be thinking about exaggeration in our learning design. And I’ve noticed that it’s a dramatic trick in popular media. But you can easily think of ways it can go wrong. So what would be appropriate exaggeration?

When I look at movies and other story-telling media (comics), the exaggeration  usually is one level.  You know, it’s like real life but some aspect is taken beyond what’s typical. So, more extreme events happen: the whacky neighbor is  maniacal, or the money problems are  potentially fatal, or the unlikely events on a trip are just more extreme.  And this works; real life is mundane, but you go too far and it treads past the line of believability. So there’s a fine line there.

Now, when we’re actually performing, whether with customers or developing a solution, it matters. It’s our  job after all, and people are counting on us.  There’s plenty of stress, because there are probably not enough time, and too much work, and…

However, in the learning situation, you’re just mimicking the real world. It’s hard to mimic the stress that comes from real life. So, I’m arguing, we should be bringing in the extra pressure through the story. Exaggerate!  You’re not just helping a customer, you’re helping the foreign ambassador’s daughter, and international relations are at stake!  Or the person you’re sweet on (or the father of said person) is watching!  This is the chance to have fun and be creative!

Now, you can’t exaggerate everything. You could add extraneous cognitive load in terms of processing if you make it too complex in the details. And you definitely don’t want to change the inherent decisions in the task and decrease the relevance of the learning. To me, it’s about increasing the meaning of the decisions, without affecting their nature. Which may require a bit of interpretation, but I think it’s manageable.

At core, I don’t think I’m exaggerating when I say exaggeration is one of your tools to enhance engagement  and effectiveness. The closer we bring the learning situation to the performance situation, the higher the transfer. And if we increase the meaningfulness of the learning context to match the performance context, even if the details are more dissimilar, I think it’s an effective tradeoff. What do  you think?

Example Diagram

by Leave a Comment

No, not a diagram that’s an example, a diagram about examples!  I created this because I needed a diagram to represent examples. I’ve written about them, and I have diagrams for other components of learning like models. However, I wanted to capture some important points about examples. So here we go.

Example elements

The idea here is that an example should be a story, with narrative flow. You start with a problem, and flow through the process to the outcome.

One of the important elements along the way is showing the steps  and the  underlying thinking. Experts may be saying “you do this, then this” but what they’re not articulating is important to. It’s more like “I could’ve done this  or this, but because of this…” and that needs to be heard.

Even better if a mistake was made, caught, and remedied. Showing that, and how, you monitor performance as you go is important for learners to see. That’s not illustrated here, because it  is optional.

What is captured here is that there is (or should be) a conceptual model guiding your performance, and that should be explicitly referenced in the thinking. It should show how the model was instantiated because of the context, and how it led to the outcome.

These, I argue, are important points about examples that are reflected in the work of Schoenfeld as captured in Cognitive Apprenticeship (by Collins & Brown). Making thinking visible is an important component of learning whether classroom or workplace. So, have I shown  my thinking?

Labels and roles

by Leave a Comment

I was just reflecting on the different job labels there are. Some of the labels are trendy, some are indicative, but there’s potentially a lot of overlap. I’m not sure what to do about it, but I thought I’d explore it.  So this is a bit of an unconstructed thought…

To start  somewhere, let’s start with Learning Architect. This is an interesting one (and one I just chose on a project). It leverages the metaphor of the relationship between an architect and the contractor who builds it. The architect imagines the flow of people, places of rest, and creatively evaluates how to match the requirements with the available space (and budget). Then someone else builds it. This is similar to a learning designer, who envisions a learning experience via a storyboard (mapping to a blueprint), before handing off to a developer.

So what is a learning experience designer?  Here is someone envisioning the cognitive (and aesthetic) flow the learner will go through.  It’s looking at addressing the change in knowledge and emotions, as a user experience designer might for an interface.  Whether they build it or not implies they’re a learning experience developer instead/in addition.

Right now I see both as equivalent. An architect is developing the flow of people and their emotions in the space. Where do you want them active, and where do you want them reflective?  The learning experience designer similarly. Are they just different cuts on the same role? I note that in the 70:20:10 process of Arets, Jennings, and Heijnen, learning architect is a role that sits between doing the analysis and implementing the solution.

I also have heard of a learning  strategist.  This could be the same, coming up with a series of tactics to transform the learner into someone with new capabilities.  Or this could be a meta-level, a role I frequently play, reviewing the design process for changes that can maximize the outcomes with a minimum of disruption.

Then there’s learning  engineering, which is in the process of being defined by a committee.  It not only includes the learning science of design, but the technical implementation. Certainly architects and designers need to be aware of the tech, not stipulating the impossible, but this role goes deeper, on to systems integration and more.

Of course, we have the traditional instructional designer, which captures the notion of facilitated learning, but not the integration of the aesthetic component.  And, on the whole, I’m avoiding the ‘developer’ label, as the people who take storyboard to an realized experience.  There are clearly people who have to straddle both (I recently asked an audience how many were sole practitioners in this sense, and a majority seemed to have to design and develop (presumably in the tools that support that).

All these labels may reflect how an organization is dividing up the whole process. I’m not even certain that the way I’ve characterized them is accurate.  What labels am I missing? What nuances?  Does this make sense?