One of the things I’ve recognized is that we don’t pay enough attention to context. It turns out to be a really important factor in cognition, as our long-term memory interacts with the current context to determine our interpretation. And, as such, makes our interpretations very ’emergent’. Thus, our training needs to ensure that we’re liable to make the right interpretation and so choose the right action. Do we do this well? And can artificial intelligence (AI), specifically generative AI (GenAI), help? Here’re some thoughts on context and models.

So, we’ve gone from symbolic models to sub-symbolic ones as we’ve moved to a ‘post-cognitive’ interpretation of our thinking. What’s been realized is that we’re not the formal logical reasoning beings that we’d like to think. Instead, we’re very much assembling our understanding on the fly as an interaction between context and memory. In fact, our emergent memory can be altered by the context, as Beth Loftus’ research demonstrated. Which means that, if we want specific interpretations and reactions (e.g. making decisions under uncertainty), we should be careful to ensure that we provide training across a suitable suite of contexts.

Now, active inference models of cognition suggest that we’re actively building models of how the world works. Thus, we’re abstracting across experiences to generate ever-more accurate explanations. Research on mental models suggests that they’re incomplete, not completely accurate, and, arguably most importantly, hard to get rid of if they’re wrong. Thus, providing good models beforehand is important, and work by John Sweller further suggests that examples showing models in context benefit learning. You can present the model, but ultimately the learner must ‘own’ it. So, it’s important to know the models and their range of applicability to facilitate that abstraction.

What is important to know, however, is that GenAI doesn’t build models of the world. This was an important (and, sadly, not self-generated) realization for me. The implication, however, is clear. I have maintained that GenAI can’t understand context, and thus can’t generate suitable practice environments. Which, of course, is to the good for designers, since it leaves them a role ;). Importantly, however, this framing also suggests that GenAI also can’t choose an appropriate suite of contexts for practice, since it doesn’t understand models and how they’re applicable (and when not). (Another designer role!)

I am all for using technology to complement our own cognition. However, that entails knowing what the true affordances of the technology are, and also what it can’t do. So, GenAI can help think of great settings for practice. Along with a person (an expert actually) to vet the suggestions, of course. It can think of things we might forget, or ones we haven’t thought of yet. It can, of course, also create ones that aren’t realistic. There’re potentially great opportunities, but we have to know what matters, and what doesn’t. Context and models matter. GenAI can’t understand them. You can take it from there.

For reasons, I’ve been looking at multiple-choice questions (MCQs). Of course, for writing them right, you should look to Patti Shank’s book Write Better Multiple-Choice Questions.  And there’s clearly a need!  Why? Because when it comes to writing meaningful MCQs, I’m wanting to move us from knowledge to performance. And the vast number of questions I found didn’t do that.

To start, I’ll point, as I often do, to Pooja Agarwal’s research (plays to my bias ;). She found that asking high-level questions (e.g. application questions, or mini-scenarios as I like to term them) leads to ability to answer high-level questions (e.g., to do). What wasn’t necessary were low-level knowledge questions. She tested low alone, high alone, and low + high. What she found that was to pass high tests, you needed high questions. Further, low questions didn’t add anything. I’ll also suggest that our needs, for our learners and our organizations, are the ability to apply knowledge in high-level ways.

Yet, when I look at what’s out there, I continually see knowledge questions. They violate, btw, many principles of good multiple questions (hence Patti’s book ;). These questions often have silly or obvious alternatives to the right answer. They include the wrong length responses, and too many (3 is ideal, usually, including the right answer). We also see a lack of feedback, just ‘right’ or ‘wrong’, not anything meaningful. We also see too many questions, or incomplete coverage, and arbitrary criteria (why 80%?). Then, too, the absolutes (never/always, etc), which isn’t the way to go. Perhaps worst, they don’t always focus on anything meaningful, but query random information that was in no way signaled as important.

Now, I suppose I can’t say that knowledge questions should be avoided. There might be reasons to ensure they’re there for diagnostic reasons (e.g. why are learners are getting this wrong). I’d suggest, however, that such questions are way overused. Moreover, we can do better. It’s even essentially easy (though not effortless).

What we have learners do is what’s critical for their effective learning, If we care (and we should), that means we need to make sure that what they do leads to the outcomes our organizations need. Which means that we need lots of practice. Deliberate practice, with desirable difficulty, spaced out over time. We need reactivation, for sure. But what we do to reactivate dictates what we’ll be able to do. If we ask people knowledge questions, they’ll be able to answer knowledge questions. But that has been shown to not lead to their ability to apply that knowledge to make decisions: solve problems, design solutions, generate better practices.

So, we can do better. We must do better. That is, if we want to actually assist our organizations. If we’re talking skilling (up-, re-, etc), we’re talking high-level questions. On the way, perhaps (and recommended), to more rigorous assessment (branching scenarios, sims, mentored practice, coaching, etc), Regardless, we want what we have learners do be meaningful, When we’re moving from knowledge to performance, it’s critical, And that’s what I believe we should be doing.

(BTW, technology’s an asset, but not a solution. As I like to say:

If you get the design right, there are lots of ways to implement it; if you don’t get the design right, it doesn’t matter how you implement it. )

We write for lots of reasons. It’s all about communication, but with different purposes, there should be different writing. Just for books, the language in a thriller should be different than for thoughtful stories. Writing for ads is different than writing for science. And, writing for learning is different than writing for other purposes. What am I talking about?

What research tells us, as Ruth Clark lets us know, is that we learn better from conversational language. Formal language, such as in an encyclopedia, or a textbook, doesn’t work for elearning or how an instructor talks to an audience. You want to be informal, personal, and more. Yet too often our prose is tedious.

Dialog, in particular, should be authentic to the speaker. I quail when I see characters spouting language straight out of an instructional manual or, worse, a marketing spiel. Good character development goes beyond stereotypes and develops some personality. This should come through in their language. Writing dialog, then, isn’t what most designers have been trained in. Which means that designers shouldn’t write dialog, or at least get external support whether training, even just peer review.

Writing for learning needs to be clear, of course. It also needs to be accurate. And yet, it shouldn’t be onerous to read. If there are barriers to comprehension, you’re putting in unnecessary barriers to your learning outcome. Really, you’re managing cognitive load. Obtuse language impedes processing, and learning is processing-intensive enough!

I’ve talked before about the importance of emotion in learning, for motivation, keeping anxiety under control, building confidence, and more. Writing is one of the most compact forms of media for communicating, and so we want our language to address these issues as well. Conversational language helps reduce anxiety by being familiar, and shows relatedness, part of the Self-Determination Theory of motivation. When folks believe we care about them, they’re more inclined to succumb to our ministrations.

Writing for learning is one of the elements necessary for the appropriate use of media. We should use the right media for the message (with a caveat about the value of novelty), and then we should apply the right media correctly. That is, ensuring we apply the appropriate expertise. We can make changes, such as my common example of Ken Burn’s compelling use of still images in his video documentary of the Civil War, but even then there are accommodations. In short, writing for learning has some particular constraints, and we as designers should be aware of them.

There’s more, of course. What you write in an introduction is different than what’s presented about a model, than the narrative for an example, for the instructions versus the description of the context for retrieval practice, etc. Knowing what the role is, and the appropriate writing, becomes habit with experience, but like all learning, models and feedback help accelerate the path there. You need to know not just what to write, but how and when. Those are my thoughts, what are yours?

When we do learning, we must be active. That is, it’s not enough to receive information. (Unless we’re actively practicing and attending presentations are reflection.) We must do! Then the question becomes one of doing ‘what’? I’m seeing too many of the wrong sorts of things in play, so it’s worth asking: what sorts of activities should we be doing?

Cognitively, we need to perceive information to get it into working memory. From there, to get into long-term memory – and be useful – we need to elaborate and practice retrieval. Elaboration is the process whereby we strengthen connections between the new material and the familiar. This increases the likelihood of activation in context. Then, we need to practice retrieving the knowledge for use. This strengthens our ability to retrieve and apply as we need.

One thing to note is that research shows that we don’t need to retrieve the fact-based knowledge before practicing retrieval to actually use. Our goal for organizational learning is to use information to make meaningful decisions. Better fact-recall isn’t likely to be what will help your organization thrive. Instead, what matters is acquiring the new skills that will define the ability to adapt.

For elaboration, what we increasingly hear is about ‘generative‘ learning activities. These are when you’re taking new information, and processing it more deeply. It can involve rephrasing, visualizing, and of course connecting it to your prior experience. These activities help strengthen the information into long-term memory.

An associated task it to practice using the information. That is, putting learners into situations where they need to use the new information to make decisions that they couldn’t before. The ideal situation, of course, is mentored live practice, but…there are limitations. Individual mentoring isn’t always cost-effective. Also, live practice may have consequences for wrong answers. In many cases, we use simulations. These can be programmed, or branching scenarios. Even mini-scenarios (e.g. better written multiple-choice questions) are a good option.

What we don’t need are fact-check questions. As above, there’s no real benefit. They may make us feel good, but they aren’t inclined to make us better at using the information. There are lots of bad practices around this. We can just use knowledge questions, thinking we’re helping learning (and not). Worse, I’ve seen many cases where they’re asking for arbitrary bits of information that aren’t highlighted. Also, too often we’re presenting way too much information than people can remember at one time (or at all).

So, if we’re to design effective learning, what sorts of activities is an important question. We don’t need fact-checks. We do benefit from processing, and retrieval. That’s worth practicing and performing. Review your work and look at what you’re having learners do. If it’s not elaboration and retrieval, you’re wasting learners’ time and your efforts. Why do that?