Learnlets

Secondary

Clark Quinn’s Learnings about Learning

The Wrong Bucket Lists

by 2 Comments

color bucketsOur brains like to categorize things; in fact, we can’t really  not do it. This has many benefits: we can better predict outcomes when we can categorize the situation, we can respond in appropriate ways via shared conceptualizations, and so on. It also has some downsides: stereotyping, for one. I reckon there’re tradeoffs, of course. But we also have to worry about when we over-use categorization, we can risk making the wrong bucket lists.

Our desire for simplification and categorization is manifest. The continued interest in reading one’s horoscope, for instance. And the continued success of personality typings, despite the evidence of their lack of utility. Other than the Big 5 or HEXACO, the rest are problematic at best. I’m just reading Annie Murphy Paul’s  The Cult of Personality Testing  (the predecessor to her  The  Extended Mind) and hearing abuses like Rorschach tests being used in child custody decisions is really horrific. Similarly, to hear that people are being denied employment based on their color (not race, but their ‘color’ on a particular test, blue or orange) isn’t new and continues (as does the other, sadly).  Most of these tests don’t stand up to scientific scrutiny!

This is the explanation of learning styles, too, another myth that won’t die. Generations similarly. We like to have simplification. Further, there are times it’s useful. For example, recording your blood type can prevent potentially life-threatening complications. Having a basis to adapt learning, such as people’s performance (success or failure), also. Even more so if additional factors are added, such as confidence. Yet, we can overdo it. We might over-categorize, and miss important nuances.

Todd Rose’s  The End of Average made an excellent case for not trying to conform people into one bucket. In it, he points out that when we assign a single grade for complex performance, we miss important nuances. For instance, if you get it wrong,  why did you get it wrong? It matters in terms of the feedback we might give you. If you had one misconception instead of another, you should get different feedback than if you had the other.

How do we reconcile this? There’re benefits to simplifications, and risks. We have to be careful to simplify as much as we can,  and no simpler. Which isn’t an easy task to undertake. The best recommendation I can make is to be mindful of the risks when you do simplify. Maybe start more broadly, and then winnow down? Explicitly consider the risks and costs as well as the benefits and savings. We’re using learner personas in a project. Many times, these personas can differ on important dimensions, and characterize the audience space in ways that a simple ‘the learner’ can’t capture.

Overall, we want to make sure we’re only using simplifications and categorizations in ways that are both helpful  and scrutable. When we do so, we can avoid the wrong bucket lists. That should be our goal, after all.

Sensitivities and Sensibilities

by 2 Comments

We are currently experiencing a crisis of communication. While this is true of our nation and arguably the world, it‘s also true in our little world of L&D. Recently, there have been at least four different ‘spats‘ about things. While I don‘t want to address the specifics of any of them, what I do want to do is talk about how we engage. So here‘s a post on sensitivities and sensibilities.

First, let me be clear, I‘ve some social issues. I‘m an introvert, and also miss social cues. I also have a bad habit of speaking before I‘ve done the knowledge-check: is this true, kind, and necessary? Subtlety and diplomacies aren‘t my strong suit. I continue to be a work in progress. Still, I never intentionally hurt anyone, at least not anyone who hasn‘t demonstrated a reliable propensity to violate norms that I feel are minimum. I continue to try to refine my responses.

There are two issues, to me: what we should say, and how we should say it. For instance, I think when someone says something wrong, we need to educate. Initially, we need to evaluate the reason. It could be that they don‘t know any better. Or it could be that they‘re deliberately trying to mislead.  

Let‘s also realize we‘re emotional animals. If I‘m attacked, for instance, I’m likely to blame myself, even when it’s wrongly. Others are highly unlikely to wear blame, and lash out. We are affected by our current context; we are more critical if we‘re tired or otherwise upset, and on the reverse are more tolerant if rested and content.  

I‘m also aware that we have no insight into where someone‘s coming from. We can guess, but we really don‘t know. I really learned this when I was suffering from a pinched nerve in my back; I have more sympathy now since I‘ve come to recognize I don‘t know what anyone else is living with.

So, I‘m trying to come up with some principles about how to respond. For instance, when I write posts about things I think are misguided or misleading, I call out the problems, but not the person, e.g. I don‘t link to the post. I‘m not trying to shame anyone, and instead want to educate the market. I think this is a general principle of feedback: don‘t attack the person, attack the behavior.  

Also, if you‘re concerned about something, ask first. Assume good intentions. How you ask matters as well. The same principle above applies: ask about the behavior. I’m  impressed with those who worry about the asker. If the ask seems a bit harsh, they wonder whether the asker might be struggling. That‘s a very thoughtful response.  

There‘s a caveat on all this: if folks continue to promote something that‘s demonstrably wrong, after notification, they should get called out. Here in the US, the first amendment says we can say whatever, but it doesn‘t say we don‘t have any consequences from what we say. (You can‘t yell ‘fire‘ in a crowded theatre if there isn‘t one!) Similarly, if you continue to promote, say, a debunked personality test, you can be called out. ;)

So this is my first draft on sensitivities and sensibilities. Assume good intent. Ask first. Educate the individual and the market. Don‘t attack the person, but the behavior. I‘m sure I‘m missing situations, conditions, additional constraints, etc. Let me know.  

Confidence and Correctness

by Leave a Comment

Not surprisingly, I am prompted regularly to ponder new things. (Too often, in the wee hours of the morning…) In this case, I realize I haven’t given a lot of thought to the role of confidence  (PDF). It’s a big thing in the system my co-author on an AI and ID paper, Markus Bernhardt, represents, so I realized it’s time to think  about it some more. Here are some thoughts on confidence and correctness.

Confidence by correctnessThe idea is that it matters whether you get it right, or not, and whether you’re confident, or not. That is, they interact (creating the familiar four quadrant model). You can be wrong and unconfident (lo/no), wrong and confident (hi/no),  right and unconfident (lo/yay), and right and confident (hi/yay). Those are arguably importantly different. In particular for what they imply about what sort of intervention makes sense.

I was pondering what this suggests for interventions. I turned it 90 degrees to the left, to put low/no to the left, or beginning spot, and hi/yay to the right, and the other two in-between.  Simplified, my view is that if you’re wrong and not confident, you don’t know it. If you’re wrong and believe you know it, you’re at a potential teachable moment. When you’re right, but not confident, you’re ready for more practice. If you’re right and confident, it may be time to move on.

Which suggests, looking back at my previous exploration of worked examples, that the very first thing to do is to provide worked examples if they’re new. At some point, you give them practice. If they get it right but aren’t confident, you give more practice at roughly the same level. If they’re wrong but confident, you give them feedback (and arguably ramp them backwards). Eventually they’re getting it right  and confident, and at that point you move on (except for some spaced and varied reactivation).

Assessing confidence is an extra step, but there seems to be a valid reason to incorporate it in your learning design. The benefits of being able to more accurately target your interventions, at least in an adaptive system, suggest that the effort is worth it. That’s my initial thinking on confidence and correctness. What’s yours?

My Personal Knowledge Management Approach

by Leave a Comment

Last week, in our Learning Development Accelerator You Oughta Know session, we had Harold Jarche as a guest. Harold’s known for many things, but in particular his approach to continual learning. Amongst the things he shared was a collection of others’ approaches. I checked and I hadn’t made a contribution! So with no further ado, here’s my personal knowledge management approach.

First, Harold’s Personal Knowledge Management (PKM) model has three components: seek, sense, and share. Seeking is about information coming in, that is, what you’re looking for and the feeds you track. It can be in any conceivable channel, and one of the important things is that it’s  your seeking. Then, you make sense of what comes in, finding ways to comprehend and make use of it. The final step is to share back out the sense you’ve made. It’s a notion of contributing back. Importantly, it’s not that necessarily anybody consumes what you share, but the fact that you’ve prepared it for others is part of the benefit you receive.

Seek

Most seeking is two-fold, and mine’s no exception. First of all there’s the ‘as needed’ searches for specific information. Here I typically use DuckDuckGo as my search engine, and often end up at Wikipedia. With much experience, I trust it.  If there are multiple hits and not a definitive one, I’ll scan the sources as well as the title, and likely open several. Then I review them until I’m happy.

The second part is the feeds. I have a number of blogs I’m subscribed to. There are also the people I follow on Twitter. On LinkedIn, a while ago I actively removed all my follows on my connections, and only retained ones for folks I trust. As I add new people, I similarly make a selection of those I know to trust, and ones who look interesting from a role, domain, location, or other diversity factor.  An important element is to be active in selecting feeds, and even review your selections from time to time.

Sense

Sometimes, I’m looking for a specific answer, and it gets put into my work. Other times, it’s about processing something I’ve come across. It may lead me to diagramming, or writing up something, frequently both (as here). Diagramming is about trying to come to grip with conceptual relationships by mapping them to spatial ones. Writing is about creating a narrative around it.

Another thing I do is apply knowledge, that is put it into action. This can be in a design, or in writing something up. This is different than just writing, for me. That is, I’m not just explaining it, I’m using it in a solution.

Share

To share, I do things like blog, do presentations and workshop, and write books. I also write articles, and sometimes just RT. Harold mentioned, during the session, that sharing should be more than just passing it on, but also adding value. However, I do sometimes just like or share things, thinking spreading it to a different audience is value. If you’re not too prolific in your output, I reckon that the selected shares add value. Of course, in general if I pass things on I do try to make a note, such as when sharing someone else’s blog that I thought particularly valuable.

So that’s my process. It’s evolving, of course. We talked about how our approaches have changed; we’ve both dropped the quantity of posts, for instance. We’re also continually updating our tools, too. I’ve previously noted how comments that used to appear on my blog now appear on LinkedIn.

To be fair, it’s also worth noting that this approach scales. So workgroups and communities can do a similar approach to continually processing. Harold’s done it in orgs, and it factors nicely into social learning as well. One attendee immediately thought about how it could be used in training sessions!

So that’s a rough cut at my PKM process. I invite you to reflect on yours, and share it with Harold as well!

I discuss PKM in both my Revolutionize L&D book, and my Learning Science book.

Emphasis and Effort

by 1 Comment

For reasons that aren’t quite clear (even to me), I was thinking about where, on a continuum, do L&D elements fit? Where does performance support go? Formal learning? Informal learning? I began to think that it depends on what focus you’re thinking of. So here’re some nascent thoughts on emphasis and effort.

To start with, I generally think of formal learning as the starting point. For instance, in thinking about performance & development (as an alternative to learning & development), I put training first. Similarly, in my strategy work, I likewise suggest the first step is to put learning science more central in training. Here, the order is:

  • Formal Learning
  • Performance Support
  • Informal Learning

I’m looking as much as where we typically start. This may well be because training is always the line of first-response (throw training at it!). Also perhaps because it’s familiar (it looks like school).

However, in another cut at it, I started with performance support. Here, I was thinking more about the utility to achieve goals rather than the way L&D allocates resources. That is, from a performer’s perspective, if the answer can be in the world, it should. I can use a tool to achieve my goal rather than have to take a course. Still, taking a pre-digested course is easier than having to work together to collaborate and solve it. Of course, if someone else has the answer, just asking and getting it is easier than working to create an unknown answer. (So, do I need to separate out communication from collaboration? Hmm…) Thus, the list here might be:

  • Performance Support
  • Formal Learning
  • Informal Learning

However, if I look at it from the effort required from L&D, a new order emerges. Here, formal learning is hardest. That is, if you’re doing it right. To successfully get a persistent change in the way someone does something is harder than even facilitating informal learning, and performance support is easiest. Not saying that any are trivial, mind you, designing good job aids isn’t easy, it’s just not as hard as designing a whole course. Then the list comes out like this:

  • Performance Support
  • Informal Learning
  • Formal Learning

I guess there isn’t one answer. To do this successfully, however, requires an understanding how to do all of the above, and then apply as priorities demands. If you’ve expert performers, you’ll do something different than if you have high turnover. If you’re doing something complex, your design strategies may differ from something important. However, you do need to know the tradeoffs in emphasis and effort to make the right calls. Am I missing something important here?

Working with SMEs

by 1 Comment

In a recent post, I talked about how expertise is compiled away, and the impact on designing learning and documentation. Someone, of course, asked  how do you then work with SMEs to get the necessary information. Connie Malamed, one of our recognized research translators, has recently written about getting tacit knowledge, but I also want to address the more usual process.  I thought I’d written about it somewhere, but I can’t find it. So here are some thoughts on working with SMEs.

First, I’ve heard from several folks experienced in this that any one SME may not have both necessary elements. One element is to have a good model to guide the performance. The second elements is the ability to articulate that model. Their solution is to work with SMEs in groups. Guy Wallace (Eppic), Roger Schank (Socratic Arts), and Norman Bier (CMU) have all mentioned to me that they’ve found utility in getting SMEs together as a group and having that knowledge negotiation unpack the necessary learnings. They’re all folks worth listening to. You have to manage the process right, of course, but if you can do it, it’s useful.

I suggest that you also want several  different types of SMEs. You want not only the top performers, and theoretical experts, you also want just-past novices (also attributable to Guy) and supervisors. Theorists can give you models, while top performers can talk about the practical implications. Novices can let you know what they found hard to understand, and supervisors provide insight into what performers typically do wrong. All are helpful information for different parts of the learning.

Another trick I use is to focus on decisions. I argue that making better decisions will be more important to organizations than the ability to recite knowledge. SMEs  do have access to all the knowledge they learned, and it’s easy for them to focus on that. That’s where you get ‘info dump’ courses and bullet point-laden slides. By using decisions as a focus, you cut through the knowledge. “What decisions will they make differently/better as a result of this knowledge?” is a helpful question.

You can use questionnaires as well. Asking specifically about the elements: models, misconceptions, consequences, can be a good preliminary step before you actually talk to them. Or have a template for content for them to fill out. Any guidance and structure helps keep them focused.

Another preparatory step is to create a draft proposal of the information. You’ll likely be getting a dump of PDFs and PPTs). Process that material, and make your first, best, guess. It’s easier to critique than generate, so if you’re willing to be wrong (and why not), you can have them shoot holes in what you did. You’ll have focused on decisions (right?), and they’ll fix it, but you’ll have biased them for action.

Of course, you want to ensure you test for confirmation. You should circulate what you have learned, and get validation. You’ll need to have clear objectives that operationalize your learnings. You then should prototype and test what you’ve developed and see if it actually changes the behavior in useful ways. Ensure that your focus actually leads to the necessary change.

There are other elements you want from SMEs, such as their personal interest. However, it’s critical that you get them to focus on behavior change. It’s not easy, but it’s part of the job. Working with SMEs, correctly, is key to designing learning experiences that address real needs. These are my thoughts, what are yours?

Experts and Explanations

by 1 Comment

blueprint pencil rulerI’ve been going through several different forms of expert documentation. As a consequence, I’ve been experiencing a lot of the problems with that! Experts have trouble articulating their thinking. This requires some extra work on the part of those who work with them, whether instructional designers, technical writers, editors, whoever. There are some reliable problems with experts and explanations that are worth reviewing.

The start of the problem is that the way we acquire expertise is to take our conscious thinking and automatize it, basically. We want to make our thinking so automatic that we no longer have to consciously think about it. So, we basically compile it away. Which creates a problem. For one, what we process into memory may not bear a close resemblance to what we have heard and applied. That is, the semantic language we use to guide our practice and internalize may not be what we store as we automate it.

It’s also the case that we lose access to that compiled away expertise. There’s evidence of this, for one from the results of research by the Cognitive Technology group at the University of Southern California showing experts can’t access about 70% of what they do! Another piece of evidence is the widespread failure of so-called ‘expert systems’ in the 80s, resulting in the AI winter. Whether the locus of the problem is in what actually gets stored, or access to it, the result is that  what we were told to do, and say we do, may not actually be close to what we actually do.

Another problem is that experts also lose touch with what they grappled with as novices. What they take for granted isn’t even on the radar of novices. So it’s difficult to get them to provide good support for acquiring skills or understanding. Their attempts at explanations for reference of instruction fail.

All told, this leads to systematic gaps in content. I’ve been seeing this manifest in explanations that may say what to do, but not why or how. There may be a lack of examples, and the thinking behind the examples I  do see isn’t there.  There’s also a lack of visual support. They’re not including diagrams when it’s conceptual relationships that need understanding. They’re also not including images when context is needed. They shouldn’t necessarily be blamed, because they don’t need the support and can’t even imagine that others do!

It’s clear that experts should not be the ones doing the explanations. They’re experts, and they have valuable input, but there needs to be a process to avoid these problems. We need tech writers, IDs, and others to work with experts to get this right. Too often we see experts being tasked with doing the explanations, and we live with the consequences.

What to do? One step is to let experts know that their expertise is in their domain, but the expertise in extracting that expertise and presenting it lies in others. To do so convincingly, you’ll need the science about why. For another, know techniques to unearth that underlying thinking. Also allow time in your schedule for this to happen. Don’t think the SME can just give you information; you’ll have to process what you get to rearrange it into something useful. You may also need some sticks and carrots.

As I wrestle with the outputs of experts, here’s my plea. There are wonderful ways experts and explanations can work out, but don’t take it for granted. Don’t give experts the job of communicating to anyone but other experts, or to experts on working with experts to get explanations. Fair enough?

Good and bad advice all in one!

by 2 Comments

I was asked to go to read an article and weigh in. First, please don’t do this if you don’t know me. However, that’s not the topic here, instead, I want to comment on the article. Realize that if you ask me to read an article, you’re opening yourself up to my opinion, good  or bad. This one’s interesting, because it’s both. Then the question is how do you deal with good and bad advice all in one.

This article is about microlearning. If you’ve been paying attention (and there’s no reason you should be), I’ve gone off on the term before. I think it’s used loosely, and that’s a problem because there are separate meanings, which require separate designs, and not distinguishing them means it’s not clear you know what you’re talking about. (If someone uses the term, I’m liable to ask which they mean! You might do the same.).

This article starts out saying that 3-5 minute videos are  not  microlearning. I have to agree with that. However, the author then goes on to document 15 points that are important about microlearning. I’ll give credit for the admission that there’s no claim that this a necessary and complete set. Then, unfortunately, I also have to remove credit for providing no data to  support the claims!  Thus, we have to evaluate each on it’s own merits.  Sorry, but I kinda prefer some sort of evidence, rather than a ‘self-evident’ fallback.

For instance, there’s a claim for brevity. I’ve liked the admonition (e.g. by JD Dillon) that microlearning should be no longer, and no shorter, than necessary. However, there’s also a claim here that it should be “3 – 10 minutes of attention span”. Why? What determines this? Human attention is complex, and we can disappear into novels, or films, or games, for hours. Yes, “Time for learning is a critical derailer”, but…it’s a factor of how important, complex, and costly if wrong the topic is. There’s no one magic guideline.

The advice continues in this frame: there’re calls for simplicity, minimalism, etc. Most of these are good principles,  when appropriately constrained. However, arbitrary calls for “one concept at a time is the golden rule”  isn’t necessarily right, and isn’t based on anything other than “our brains need time for processing”. Yes, that’s what automation is about, but to build chunks for short term memory, we have to activate things in juxtaposition. Is that one concept? It’s too vague.

However, it could be tolerated if some of the advice didn’t fall prey to fallacious reasoning. So, for instance, the call for gamification leans into “Millennials and Gen Z workforce” claims. This is a myth. Gamification itself is already dubious, and using a bad basis as an assumed foundation exacerbates the problem.  There are other problems as well. For one, automatically assuming social is useful is a mistake. Tying competition into the need to compete is a facile suggestion. Using terms like ‘horde’ and ‘herd’ actually feels demeaning to the value of community. A bald statement like “Numbers speak louder than words!” similarly seems to suggest that marketing trumps matter. I don’t agree.

Overall, this article is a mixed bag. So then the question arises, how do you rate it? What do you do? Obviously, I had to take it apart. The desire for a comment isn’t sufficient to address a complex suite of decent principles mixed up with bad advice and justified (if at all) on false premises. I have to say that this isn’t worth your time. There’s better advice to be had, including on microlearning. In general, I’ll suggest that if there’s good and bad advice all in one, it’s overall bad. Caveat emptor!

Generic Thinking Skills?

by 3 Comments

Recently, a colleague asked a few of us about our views on critical thinking skills. This is actually a contentious topic. There are broad claims of the need for them, increasingly, even showing up in job advertisements. On the other hand, researchers and others have weighed in against them, saying that expertise is the only lever. I tend to lump critical thinking skills in with the broader issue of generic thinking skills, so what are the issues?

Upfront, I’ll admit that I like the concept of generic thinking skills. Say, for instance,  learning-to-learn skills. That is, domain independent skills that lead to better approaches. It seems to make sense that, in the absence of specific knowledge, some general approaches are more useful than others. For instance, faced with a new domain, I’d be inclined to expect that systematic experimentation and observation would be better than random trial and error.

On the other hand, prominent psychologists like John Sweller and Paul Kirschner have said that domain-specific skills are the only way to bet. There is significant evidence that expertise matters in successful approaches to problem-solving, and others. While we have some innate skills for domains that are biologically primary, learning in other domains requires expertise.

Is there, then, any evidence for generic skills? Based on Micki Chi’s work on the value of self-explanation, Kate Bielaczyc and others have found that instruction on systematically explaining steps in examples help, across domains. In my own Ph.D. thesis, I trained folks on analogical reasoning skills, and found improvement (for component skills that weren’t a) already ceilinged or b) were perceived to be immutable, across different problem types.

How, then, do we reconcile these conflicting viewpoints? My (self- :) explanation is that it’s a matter of degree, a continuum rather than a dichotomy. The more domain knowledge you possess, the more likely you are to find a good answer. However, what if you’re in a new domain where you don’t have relevant expertise to hand? In that case, I’ll suggest that there are benefits to some approaches over others, and training those general skills is justifiable. That is, general skills are weaker than domain specific skills, but general skills are better than nothing!

We know that there are practices that improve outcomes. For instance, I’ve written about how to, and not to, do brainstorming. Similarly, I believe Harold Jarche’s Seek-Sense-Share model works across domains. Systematic creativity is  not an oxymoron!  That’s the story I’m holding on to about generic thinking skills. What are your thoughts on the topic?

Accreditation?

by 1 Comment

As occasionally happens, I was asked a question on LinkedIn. In this case, it was about my thoughts on accreditation. Also, as occasionally happens, I thought that I’d share my thoughts in this forum, and look for feedback to improve my thinking. So here’re some thoughts on accreditation. I welcome yours!

First, let’s be clear, I am not an expert on accreditation. I haven’t accredited anything, for one ;). I  did look into it, at one point many years ago. I’ve also served on independent board of directors or advisory boards for several entities. In the former case, we have a legal responsibility to provide guidance. In the latter case, we provide the best guidance, but of course the organization isn’t obliged to comply. The former, in particular, serves as a quality check, or a form of accreditation.

When I looked into accreditation for educational institutions, the requirement isn’t about the actual curriculum, but instead that there is a library and that there are processes for review and revision of course offerings. That is, it was about the support for learning and quality processes, not the actual offering. This creates a process support that should ensure quality, yet also the ability to apply this to institutions with a wide variety of offerings.

Institutions can also seek accreditation by organizations in particular areas of curriculum. Offerings in computer science, business, and others for instance, receive review and then can receive approval by bodies that represent the particular field. This depends on the quality of the organization doing the accreditation and their processes, of course.

There’s also accreditation on the quality of the educational process. You can also be reviewed and accredited on the basis of your pedagogy, for instance your online teaching approach. It depends, of course, on what they stipulate as quality, but that’s always going to be the case.

There are, of course, dubious accreditations. It’s not unknown for an organization or collection thereof to establish their own accrediting body that basically rubber stamps the organization(s). Caveat emptor.

In general, I think that having a scrutable external validation is a good check on quality. Whether that makes sense is probably an issue of scope. A small offering of a particular course might benefit from an independent advisory board, which provides some oversight. The larger the organization and the scope of activity, the greater the need for some external validity check.

From the other side, I think a certificate or credentials help the learner signify what they’ve accomplished.  However, without accreditation or at least a scrutable process, how do you know the skill/knowledge is appropriate and accurate? I think accreditation has the potential to be a ‘reality check’ on any offering.