You also hear about the holy trinity of engineering: cheap, fast, or good; pick two. We have real world pressures that want us to do things efficiently. For instance, we have lots of claims that generative AI will allow us to generate more learning faster. Thus, we can do more with less. Which isn’t a bad thing…if what we produce is good enough. If we’re doing good, I’ll suggest, then we can worry about fast and cheap. But doing bad faster and cheaper isn’t a good thing! Which brings us to the second issue.

What is our definition of ‘good’? It appears that, too often, good is if people ‘like’ it. Which isn’t a bad thing, it’s even the first level in the Kirkpatrick-Katzell model: asking what people think of the experience. One small problem: the correlation between what people think of an experience, and it’s actual impact, is .09 (Salas, et al, 2012). That’s zero with a rounding error! What it means is that people’s evaluation of what they think of it, and the actual impact, isn’t correlated at all. It could be highly rated and not be effective, or highly rated and be effective. Etc. At core, you can’t tell by the rating.

What should be ‘good’? The general intent of a learning intervention (or any intervention, really) is to have an impact! If we’re providing learning, it should yield a new ability to ‘do’. There are a multitude of problems here. For one, we don’t evaluate performance, so how would we know if our intervention is having an impact? Have learners acquired new abilities that are persisting in the workplace and leading to the necessary organizational change? Who knows? For another, folks don’t have realistic expectations about what it takes to have an impact. We’ve devolved to a state where if we build it, it must be good. Which isn’t a sound basis for determining outcomes.

There is, of course, a perfectly good reason to evaluate people’s affective experience of the learning. If we’re designing experiences, having it be ‘hard fun’ means we’ve optimized the engagement. This is fine, but only after, we’ve established efficacy. If we’re not having a learning impact in terms of new abilities to perform, what people think about it isn’t of use.

Look, I’d prefer us to be in the situation where perfection to be the enemy of the good! That’d mean we’re actually doing good. Yet, in our industry, too often we don’t have any idea whether we are or not. We’re not measuring ‘good’, so we’re not designing for it. If we measured impact first, then experience, we could get overly focused on perfection. That’d be a good problem to have, I reckon. Right now, however, we’re only focused on fast and cheap. We won’t get ‘good’ until we insist upon it from and for ourselves. So, let’s shall we?

The post The enemy of the good appeared first on Learnlets.

To be clear, I’m not talking about myths. Already addressed that. But is there something to be taken away? I suggested (and practiced in my book on myths in our industry), that we need to treat people with respect. I suggest that we need to:

• Acknowledge the appeal
• Also address what could be the downsides
• Then, look to the research
• Finally, and importantly, provide an alternative

The open question is whether this also applies to talking learning.

In general, when talking about trying to convince folks about why we need to shift our expectations about learning, I suggest that we need to be prepared with a suite of stories. I recognize that different approaches will work in different circumstances. So, I’ve suggested we should have to hand:

• The theory
• The data/research
• A personal illustrative anecdote
• Solicit and use one of their personal anecdotes
• A case study
• A case study of what competitors are doing

Then, we use the one we think works best with this stakeholder in this situation.

Can we put these together? I think we can, and perhaps should. We can acknowledge the appeal of the current approach. E.g., it’s not costing too much, and we have faith it’s working. We should also reveal the potential flaws if we don’t remedy the situation: we’re not actually moving any particular needle. Then we can examine the situation: here we draw upon one of the second list about approaches. Finally, we offer an alternative: that if we do good learning design, we can actually influence the organization in positive ways!

This, I suggest, is how we might approach convincing stakeholders. And, let me strongly urge, we need to! Currently there are far too many who believe that learning is the outcome of an event. That is, if we send people off to a training event, they’ll come back with new skills. Yet, learning science (and data, when we bother) tells us this isn’t what happens. People may like it, but there’s no persistent change. Instead, learning requires a plan and a journey that develops learners over time. We know how to do good learning design, we just have to do it. Further, we have to have the resources and understanding to do so. We can work on the former, but we should work on the latter, too.

The post Convincing stakeholders appeared first on Learnlets.

In our recent learning science conference, one topic that came up was about contexts. That is, I suggest the contexts we see across examples and practice define the space of transfer. We know that contextual performance is better than abstract (c.f. Bransford’s work at Vanderbilt with the Cognitive Technology Group). The natural question is how to choose contexts. The answer, I suggest, is ad hoc: choose the minimal set of contexts that spans the space of transfer. What we’re talking about is looking for a set chosen across contexts that support the best learning.

So, in talks I’ve used the diagram to say that if you choose the set of contexts represented by the ‘o’s, you’ll be unlikely to transfer to A, whereas if you choose the ‘x’s, you’re much more likely. Let me make that concrete: let’s talk negotiation (something we’re all likely to experience). If all your contexts are about vendors (e.g. ‘o’s,) you may not apply the principles to negotiating with a customer, A. If, however, you have contexts negotiating with vendors, customers, maybe even employers (‘x’s), you’re more likely to transfer to other situations. (Though your employer might not like it! ;)

The point that was asked was how to choose the set. You can be algorithmic about it. If you could measure all dimensions of transfer, and ensure you’re progressing from simple to complex along those, you’d be doing the scientific best. It might lead you to choose too many, however. It may be that you can choose a suite based upon a more heuristic approach to coverage. Here I mean picking ones that provide some substantive coverage based upon expertise (say, from your SME or supervisors of performance). I suspect that you’ll have to make your best first guess and then test to see if you’re getting appropriate transfer, regardless.

It’s important to ensure that the set is minimal. You don’t want too many contexts to make the experience onerous. So pick a set that spans the space, but also is slim. The right set will illuminating the ways in which things can vary without being too large. Another criteria is to have interesting contexts. You are, I’ll suggest, free to exaggerate them a little to make them interesting if they’re not inherently so.

You may also need some times when the context says not to use the focus here. What I mean is that while it could seem appropriate to extend whatever’s being learned to this situation, you shouldn’t. Some ideas support over-generalization, and you’ll need to help people learn where those limits are.

Note that the contexts are those across both examples and practice. So, learners will see some contexts in examples, then others in practice. It may be (if it’s complex, or infrequent, or costly) that you need to have lots of practice, and this isn’t a worry. Still, making sure you’re covering the right swatch across contexts will support achieving the impact in all appropriate situations.

I’m less aware of research on the spread of contexts for transfer (PhD topic, anyone?), and welcome pointers. Still, cognitive theory suggests that this all makes sense. It does to me, how about you?

The post <I>Across</I> Contexts appeared first on Learnlets.

So, the conference was about what’s known in learning science. We had topics about the foundations, limitations, media, myths, informal/social, desirable difficulty, applications, and assessment/evaluation. What, however, comes next? Where do you go from a foundation in learning science?

My answer is to figure out what it means! There are lots of practices in L&D that are grounded in learning science, but go from there to application. My initial list looks like this:

1. Instructional design. Knowing the science is good, but how do you put it into a process?
2. Modalities. When you’re doing formal learning, you can still do it face to face, virtually, online, or blended. What are the tradeoffs, and when does each make sense?
3. Performance consulting. We know there are things where formal learning doesn’t make sense. We want gaps and root causes to determine the right intervention.
4. Performance support. If you determine job aids are the answer, how do you design, develop, and evaluate them? How do they interact with formal learning?
5. Innovation. This could (and should; editorial soapbox) be an area for L&D to contribute. What’s involved?
6. Diversity. While this is tied to innovation, it’s a worthy topic on its own. And I don’t just mean compliance.
7. Technology. There are lots of technologies, what are their learning affordances? XR, AI, the list goes on.
8. Ecosystem. How do you put the approaches together into a coherent solution for performance? If you don’t have an ‘all singing, all dancing’ solution, what’s the alternative?
9. Strategy. There’s a pretty clear vision of where you want to be. Then, there’s where you are now. How do you get from here to there?

I’m not saying this is the curriculum for a followup, I’m saying these are my first thoughts. This is what I think follows beyond learning science. There are obviously other ways we could and should go. These are my ideas, and I don’t assume they’re right. What do you think should be the followon? (Hint: this is likely what next year’s conference will be about. ;)

The post Beyond Learning Science? appeared first on Learnlets.

I’m not going to ask this cold, of course. I’ve thought about it a bit myself, creating an initial framework (click on the image to see it larger). Ironically, considering my stance, it’s based around ADDIE. That’s because I believe the elements are right, just that it’s not a good basis for a design process. However, I do think we may need different tools for the stages of analysis, design, development, implementation, and evaluation, even if don’t invoke them in a waterfall process. I also have categories for overarching strategy, and for specific learning topics. These are spaces in which resources can reside.

There are also several different types of resources I’ve created categories for. One is an overview of the particular spaces I indicate above. Another are for information resources, that drill into a particular approach or more. These can be in any format: text or video typically. Because I’m weird for diagrams, I have them separately, but they’d likely be a type of info resource. Importantly, one is tools. Here I’m thinking performance support tools we use: templates, checklists, decision trees, lookup tables. These are the things I’m a bit focused on.

Of course, this is for evidence-based practices. There are plenty of extant frameworks that are convenient, and cited, but not well-grounded. I am looking for those tools you use to accomplish meaningful solutions to real problems that you trust. I’m looking for the ones you use. The ones that provide support for excellent execution. In addition to the things listed above, how about processes? Frameworks? Models? What enables you to be successful?

Obviously, but importantly, this isn”t done! That is, I put my first best thoughts out there, but I know that there’s much more. More will come to me (already has, I’ve already revised the diagram a couple of times), but I’m hoping more will come from you too. That includes the types of resources, spaces, as well as particular instances.

The goal is to think about the resources we have and use. I welcome you putting in, via comments on the blog or wherever you see this post, and let me know which ones you find to be essential to successful execution. I’d really like to know what L&D resources do we use. Please take a minute or two and weigh in with your top and essential tools. Thanks!

The post What L&D resources do we use? appeared first on Learnlets.

We have some of the most respected translators of learning science research to practice. Presenters are Ruth Clark, Paul Kirschner, Will Thalheimer, Patti Shank, Nidhi Sachdeva, as well as Matt Richter and myself. They’ll be providing a curated curriculum of sessions. These are admittedly some of our advisors to the Learning Development Accelerator, but that’s because they’ve reliably demonstrated the ability to do the research, and then to communicate the results of theirs and others’ work in terms of the implications for practice. They know what’s right and real, and make that clear.

The conference is a hybrid model; we present the necessary concepts asynchronously, starting later this month. Then from 11- 15 November, we’ll have live online sessions led by the presenters. These are at two different times to accommodate as much of the globe as we can! In these live sessions we’ll discuss the implications and workshop issues raised by attendees. We will record the sessions in case you can’t make it. I’ll note, however, that participating is a chance to get your particular questions answered! Of course, we’ll have discussion forums too.

We’ve worked hard to make this the most valuable grounding you can get, as we’ve deliberately chosen the topics that we think everyone needs to comprehend. I suggest there’s something there for everyone, regardless of level. We’re covering the research and implications around the foundations of learning, practices for design and evaluation, issues of emotion and motivation, barriers and myths, even informal and social learning. It’s the content you need to do right by your stakeholders.

Our intent is that you’ll leave equipped to be the evidence-based L&D practitioner our industry needs. I hope you’ll take advantage of this opportunity, and hope to see you at the Learning Science Conference 2024.

The post Learning Science Conference 2024 appeared first on Learnlets.

A colleague of ours sent me an issue of a newsletter (it included the phrase ‘make it meaningful‘ ;). In it, the author was touting a four letter acronym-based model. And, to be fair, there was nothing wrong with what the model stipulated. Chunking, maintaining attention, elaboration, and emotion are all good things. What bothered me was that these elements weren’t sufficient! They covered important elements, but only some. If you just took this model’s advice, you’d have somewhat more memorable learning, but you’d fall short on the real potential impact. For instance, there wasn’t anything there about the importance of contextualized practice nor feedback. Nor models, for that matter!

I’m not allergic to n letter acronym models. For instance, I keep the coaster I was given for Michael Allen’s CCAF on my desk. (It’s a nice memento.) His Context-Challenge-Activity-Feedback model is pretty comprehensive for the elements that a practice has to have (not surprisingly). However, learning experiences need more than just practice, they need introductions, and models, and examples and closings as well as practice. And while the aforementioned elements are necessary, they’re not sufficient. Heck, Gagné talked about nine elements.

What I realize as I reflect is that I like models that have the appropriate amount of complexity for the level of description they’re talking about. Yet I’ve seen far too many models that are cute (some actually spell words) and include some important ideas but they’re not comprehensive for what they cover. The problem, of course, is that you need to understand enough to be able to separate the wheat from the chaff. I’ll suggest to look to vetted models, that are supported by folks who know, and there are criticisms and accolades to accompany them. Read the criticisms, and see if they’re valid. Otherwise, the model may be useful.

Ok, one other thing bothered me. This model supposedly has support from neuroscience. However, as I’ve expressed before, there have yet to be results that aren’t already made from cognitive science research. This, to me, is just marketing, with no real reason to include it except to try to make it more trendy and appealing. A warning sign, to me at least.

Look, designing for learners is complex. Good models help us handle this complexity well. Bad ones, however, can mislead us into only paying attention to particular bits and create insufficient solutions. When you’re looking at simple models and complex problems, you need to keep an eye out for help, but maybe it needs to be a jaundiced eye.

The post Simple Models and Complex Problems appeared first on Learnlets.

To start, the notion is that it’s learning at the moment of need. Which sounds good. Yet, do we really need learning? The idea Jay pointed to in his book Informal Learning, was talking about Gloria Gery’s work on helping people in the moment. Which is good! But is it learning? Gloria was really talking about performance support, where we’re looking to overcome our cognitive limitations. In particular, memory, and putting the information into the world instead of in the head. Which isn’t learning! It’s valuable, and we don’t do it enough, but it’s not learning.

Why? Well, because learning requires action and reflection. The latter can just be thinking about the implications, or in Harold Jarche’s Personal Knowledge Mastery model, it’s about experimenting and representing. In formal learning, of course, it’s feedback. I’ve argued we could do that, by providing just a thin layer on top of our performance support. However, I’ve never seen same!  So,  you’re going to do, and then not learn. Okay, if it’s biologically primary (something we’re wired to learn, like speaking), you’re liable to pick it up over time, but if it’s biologically secondary (something we’ve created and aren’t tuned for, e.g. reading) I’d suggest it’s less likely. Again, performance is the goal. Though learning can be useful to support comprehending context and  making complex decisions, what we’re good at.

What is problematic is the notion of workflow and reflection in conjunction. Simply, if you’re reflecting, you’re by definition out of the workflow! You’re not performing, you’re stopping and thinking. Which is valuable, but not ‘flow’. Sure, I may be overly focused on workflow being in the ‘zone’, acting instead of thinking, but that, to me, is really the notion. Learning happens when you stop and contemplate and/or collaborate.

So, if you want to define workflow to include the reflection and thoughtful work, then there is such a thing. But I wonder if it’s more useful to separate out the reflection as things to value, facilitate, and develop. It’s not like we’re born with good reflection practices, or we wouldn’t need to do research on the value of concept mapping and sketch noting and how it’s better than highlighting. So being clear about the phases of work and how to do them best seems to me to be worthwhile.

Look, we should use performance support where we can. It’s typically cheaper and more effective than trying to put information into the head. We should also consider adding some learning content on top of performance support in times where people knowing why we’re doing it as much as what we should do is helpful. Learning should be used when it’s the best solution, of course. But we should be clear about what we’re doing.

I can see arguments why talking about workflow learning is good. It may be a way to get those not in our field to think about performance support. I can also see why it’s bad, leading us into the mistaken belief that we can learn while we do without breaking up our actions. I don’t have a definitive answer to “is workflow learning a myth” (so this would be an addition to the ‘misconceptions’ section of my myths book ;). What I think is important, however, is to unpack the concepts, so at least we’re clear about what learning is, about what workflow is, and when we should do either. Thoughts?

The post Is “Workflow Learning” a myth? appeared first on Learnlets.

What are the essential elements of feedback? Well, it should be on the performance, not the individual. It should be model-based, in that you should be using models to explain how to perform, showing examples of the model being used in context, and then asking the learner to use them. The feedback, then, uses the model to explain why what went right, or what went wrong. Also, it should be minimal other than that.

So, here I tried to show that the individual (or group, hmm) produces output. That output is evaluated by the model to ascertain correctness, or not. (Not the individual!) If the answer’s wrong, you say why, and then the right answer. If it’s right, you just reinforce the right answer.

Of course, this representation doesn’t convey the minimal aspect. It’s also not clear about using the model in the feedback. Still, so far it’s a representation I can talk to. So, this is my first stab at diagramming feedback. I welcome same!

The post Diagramming Feedback appeared first on Learnlets.

So, let’s start with myths. These, I claim, are things that have been shown not to have value by empirical research. There are studies that have examined these claims, and found them to not have data to support them. For instance, accommodating learning styles is a waste. Yes, we know people differ in learning, but we don’t have a reliable base. Moreover, people’s choices to work for (or against) their style don’t make a difference in their learning. Some of the instruments are theoretically flawed as well as psychometrically invalid.

What’s the harm? I’ll suggest several ways in which myths harm us. For one, they can cause people to spend resources (money & time)  addressing them that won’t have an impact. It’s a waste! We can also characterize people in ways that limit them; for instance if they think they learn in a particular way, they may avoid a topic or invest effort in an inappropriate way to learn it. Investing in unproven approaches also perpetuates them, propagating the beliefs to others.

Superstitions, as I define them, are beliefs nobody would claim to believe, yet somehow persist in our practices. For instance, few will claim to believe that telling is sufficient to achieve behavior change. Yet, we continue to see information presentation and knowledge test, such as “awareness” training. Why? This is a waste of effort. There aren’t outcomes from these approaches. Typically, they are legacies of expectations from previous decades, yet business practices haven’t been updated. Still, to the extent that we continue these practices, even while decrying them, we’re again wasting time and money. Maybe we tick boxes and make people happy, but we can (and should) do better.

The final category is misconceptions. These are beliefs that some hold, and others decry. They aren’t invalid, but they only make sense in certain circumstances. I suggest that those who defy them don’t have the need, and those who tout them are in the appropriate circumstance. What matters is understanding when they make sense, and then using them, or not, appropriately. If you avoid them when they make sense, you may make your life harder. If you adopt them when they’re not appropriate,  you could make mistakes or waste money.

At the end of the day, the damage done is the cost of wasting money and time. Understanding the choices is critical. To do so best, you can and should understand the underlying cognitive and learning sciences. You should also track the recognized translators of research into practice who can guide you without you having to read the original academese. To be professional in our practice, we need to know and use what’s known, and avoid what’s dubious. Please!

The post The Damage Done appeared first on Learnlets.