Archives for July 2019

Direct Instruction and Learning Experience Design

30 July 2019 by Clark Leave a Comment

After my previous article on direct instruction versus guided discovery, some discussion mentioned Engelmann’s Direct Instruction (DI). And, something again pointed me to the most comprehensive survey of educational effects. So, I tracked both of these down, and found some interesting results that both supported, and confounded, my learning. Ultimately, of course, it expanded my understanding, which is always my desire. So it’s time to think a bit deeper about Direct Instruction and Learning Experience Design.

Engelmann’s Direct Instruction is very scripted. It is rigorous in its goals, and has a high amount of responses from learners. Empirically, DI has great success, with some complaints about lack of teacher flexibility. It strikes me as very good for developing core skills like reading and maths. I was worried about the intersection of many responses a minute and more complex tasks, though it appears that’s an issue that has been addressed. I couldn’t find the paper that makes that case, however.

Another direction, however, proved fruitful. John Hattie, an educational researcher, collected and conducted reviews of 800+ meta-analyses to look at what worked (and didn’t) in education. It’s a monumental work, collected in his book Visible Learning. I’d heard of it before, but hadn’t tracked it down. It was time.

And it’s impressive in breadth and depth. This is arguably the single most important work in education. And it opened my eyes in several ways. To illustrate, let me collect for you the top (>.4) impacts found, which have some really interesting implications:

Reciprocal teaching (.74)
Providing feedback (.72)
Teaching student self-verbalization (.67)
Meta-cognition strategies (.67)
Direction instruction (.59)
Mastery learning (.57)
Goals-challenging (.56)
Frequent/effects of testing (.46)
Behavioral organizers (.41)

Reciprocal teaching and meta-cognition strategies coming out highly, a great outcome. And of course I am not surprised to see the importance of feedback. I have to say that I was surprised to see direct instruction and mastery learning coming out so high. So what’s going on? It’s related to what I mentioned in the afore-mentioned article, about just what the definition of DI is.

So, Hattie says: …”what the critics mean by direct instruction is didactic teacher-led talking from the front…” And, indeed, that’s my fear of using the label. He goes on to point out the major steps of DI (in my words):

Have clear learning objectives: what should the learner be able to do?
Clear success criteria (which to me is part of 1)
Engagement: an emotional ‘hook’
A clear pedagogy: info (models & examples), modeling, checking for understanding
Guided practice
Closure of the learning experience
Reactivation: spaced and varied practice

And, of course, this is pretty much everything I argue for as being key to successful learning experience design. And, as I suspected, DI is not what the label would lead you to believe (which I do think is a problem). As I mentioned in a subsequent post, I’ve synthesized my approach across many elements, integrating the emotional elements along with effective education practice (see the alignment). There’s so much more here, but it’s a very interesting result. Direct Instruction and Learning Experience Design have a really nice alignment.

And a perfect opportunity to remind you that I’ll be offering a Learning Experience Design workshop at DevLearn, which will include the results of my continuing investigation (over decades) to create an approach that’s doable and works. Hope to see you there!

Drink your own champagne?

25 July 2019 by Clark 1 Comment

I was talking with a vendor of a robust suite of tools. In the course of it, in my usual teasing way, I asked a question. And, while I wasn’t surprised at the answer, I was ‘concerned’. And so should you be. So I’m going to suggest you start asking of your vendors “Do you drink your own champagne?”

So, this was a manufacturer of an LMS (and some other, related, platforms). And they market their advanced capabilities. And, really, I have no problem with their tools; they seem pretty enlightened. So I asked whether they used their own tools.

And there are reasons to do so. For one, to have credibility, for sure. And, to truly know your own product. But the really important reason is to be able to understand the use experience and tune accordingly. Customer research is an important tool here as well, but it’s not the only one. You really need to use something to truly know what works and what doesn’t.

It’s a form of experimentation. I test myself by trying to apply my principles in my endeavors, and then try to take on new situations to try out my beliefs more broadly. And so should you, at the individual and organizational level. Using your own tool is a form of this. It is, essentially, testing your theory with research!

And I think most folks with services and such are likely to practice what they preach. And that could be for some bad things as well as good (thinking: learning styles vendors). But I was surprised when the answer was only “somewhat”. That’s not really good enough.

So, I’m going to suggest that this should be a question you ask of every vendor. I’m also going to suggest every vendor ensure that they do use their own tool. Internally, for their own work. Whether it’s authoring tools, a course management system, a portal, a web meeting tool, what have you. If you don’t drink your own champagne, you’re not only undermining trust, you’re losing a valuable source of information. Now, pass the bubbly, would you?

Theory or Research?

17 July 2019 by Clark Leave a Comment

There’s a lot of call for evidence-based methods (as mentioned yesterday): L&D, learning design, and more. And this is a good thing. But…do you want to be basing your steps on a particular empirical study, or the framework within which that study emerged? Let me make the case for one approach. My answer to theory or research is theory. Here’s why.

Most research experiments are done in the context of a theoretical framework. For instance, the work on worked examples comes from John Sweller’s Cognitive Load theory. Ann Brown & Ann-Marie Palincsar’s experiments on reading were framed within Reciprocal Teaching, etc. Theory generates experiments which refine theory.

The individual experiments illuminate aspects of the broader perspective. Researchers tend to run experiments driven by a theory. The theory leads to a hypothesis, and then that hypothesis is testable. There are some exploratory studies done, but typically a theoretical explanation is generated to explain the results. That explanation is then subject to further testing.

Some theories are even meta-theories! Collins & Brown’s Cognitive Apprenticeship (a favorite) is based upon integrating several different theories, including the Reciprocal Teaching, Alan Schoenfeld’s work on examples in math, and the work of Scardemalia & Bereiter on scaffolding writing. And, of course, most theories have to account for others’ results from other frameworks if they’re empirically sound.

The approach I discuss in things like my Learning Experience Design workshops is a synthesis of theories as well. It’s an eclectic mix including the above mentioned, Cognitive Flexibility, Elaboration, ARCS, and more. If I were in a research setting, I’d be conducting experiments on engagement (pushing beyond ARCS) to test my own theories of what makes experiences as engaging and effective. Which, not coincidentally, was the research I was doing when I was an academic (and led to Engaging Learning). (As well as integration of systems for a ubiquitous coaching environment, which generates many related topics.)

While individual results, such as the benefits of relearning, are valuable and easy to point to, it’s the extended body of work on topics that provides for longevity and applicability. Any one study may or may not be directly applicable to your work, but the theoretical implications give you a basis to make decisions even in situations that don’t directly map. There’s the possibility to extend to far, but it’s better than having no guidance at all.

Having theories to hand that complement each other is a principled way to design individual solutions and design processes. Similarly for strategic work as well (Revolutionize L&D) is a similar integration of diverse elements to make a coherent whole. Knowing, and mastering, the valid and useful theories is a good basis for making organizational learning decisions. And avoiding myths! Being able to apply them, of course, is also critical ;).

So, while they’re complementary, in the choice between theory or research I’ll point to one having more utility. Here’s to theories and those who develop and advance them!

Direct Instruction or Guided Discovery

16 July 2019 by Clark Leave a Comment

Recently, colleague Jos Arets of the 70:20:10 institute wrote a post promoting evidence-based work. And I’m a big fan, both of his work and the post. In the post, however, he wrote one thing that bugs me. And I realize I’m flying in the face of many august folks on whether to promote direct instruction or guided discovery. So let me explain myself ;).

It starts with a famous article by noted educational researchers Paul Kirschner, John Sweller, and Richard Clark. In it, they argue against “constructivist, discovery, problem-based, experiential, and inquiry-based teaching”. That’s a pretty comprehensive list. Yet these are respected authors; I’ve seen Richard Clark talk, have talked with John Sweller personally, and have interacted with Paul Kirschner online. They’re smart and good folks committed to excellent work. So how can I quibble?

First, it comes from their characterization of the opposition as ‘minimally guided’.Way back in 1985, Wallace Feurzig was talking about ‘guided discovery’, not pure exploration. To me, that’s a bit of a ‘straw man’ argument. Not minimally guided, but appropriately guided, would seem to me to be the appropriate approach.

Further, work by David Jonassen for one, and a meta-analysis conducted by Stroebel & Van Barneveld for another, suggested different outcomes. The general outcome is problem-based (as one instance being argued against) doesn’t yield quite as good performance on a subsequent test, but is retained longer and transfers better. And those, I suggest, are the goals we should care about. Similarly, research supports attempting to solve problems even if you can’t before you learn.

And I worry about the phrase “direct instruction”. That easy to interpret as ‘information dump and knowledge test’; it sounds like the old ‘error-free learning’! I’m definitely not accusing those esteemed researchers of implying that, but I am afraid that under informed instructors could take that implication. It’s all too easy to see too much of that in classrooms. Teacher strategies tend to ignore results like spaced, varied, and deliberate practice. Similarly, the support for students to learn effective study skills is woeful.

Is there a reconciliation? I suggest there is. Professors Kirschner, Sweller, & Clark would, I suggest, expect sufficient practice to a criteria, and that the practice should match the desired performance. I suspect they want learners solving meaningful problems in context, which to me is problem-based learning. And their direct instruction would be targeted feedback, along with models and examples. Which is what I strongly suggest. The more transfer you need, however, the broader contexts you need. Similarly, the more flexible application required would suggest the gradual removal of scaffolding.

So I really think that guided exploration, and meaningful direct instruction, will converge in what eventuates in practice. Look, insufficiently guided practice isn’t effective, and I suspect that they wouldn’t suggest that bullet points are effective instruction. I just want to ensure that we focus on the important elements, e.g. what we highlighted in the Serious eLearning Manifesto. There is a reason to think that direct instruction or guided discovery isn’t the dichotomy proposed, I’ll suggest. FWIW.

Dimensions of difficulty

11 July 2019 by Clark 1 Comment

As one of the things I talk about, I was exploring the dimensions of difficulty for performance that guide the solutions we should offer. What determines when we should use performance support, automate approaches, we need formal training, or a blend, or…? It’s important to have criteria so that we can make a sensible determination. So, I started trying to map it out. And, not surprisingly, it’s not complete, but I thought I’d share some of the thinking.

So one of the dimensions is clearly complexity. How difficult is this task to comprehend? How does it vary? Connecting and operating a simple device isn’t very complex. Addressing complex product complaints can be much more complex. Certainly we need more support if it’s more complex. That could be trying to put information into the world if possible. It also would suggest more training if it has to be in the head.

A second dimension is frequency of use. If it’s something you’ll likely do frequently, getting you up to speed is more important than maintaining your capability. On the other hand, if it only happens infrequently, it’s hard to try to keep it in the head, and you’re more likely to want to try to keep it in the world.

And a third obvious dimension is importance. If the consequences aren’t too onerous if there are mistakes, you can be more slack. On the other hand, say if lives are on the line, the consequences of failure raise the game. You’d like to automate it if you could (machines don’t fatigue), but of course the situation has to be well defined. Otherwise, you’re going to want lot of training.

And it’s the interactions that matter. For instance, flight errors are hopefully rare (the systems are robust), typically involve complex situations (the interactions between the systems mean engines affect flight controls), and have big consequences! That’s why there is a huge effort in pilot preparation.

It’s hard to map this out. For one, is it just low/high, or does it differentiate in a more granular sense: e.g. low/medium/high? And for three dimensions it’s hard to represent in a visually compelling way. Do you use two (or three) two dimensional tables?

Yet you’d like to capture some of the implications: example above for flight errors explains much investment. Low consequences suggest low investment obviously. Complexity and infrequency suggest more spacing of practice.

It may be that there’s no one answer. Each situation will require an assessment of the mental task. However, some principles will overarch, e.g. put it in the world when you can. Avoiding taxing our mental resources is good. Using our brains for complex pattern matching and decision making is likely better than remembering arbitrary and rote steps. And, of course, think of the brain and the world as partners, Intelligence Augmentation, is better than just focusing on one or another. Still, we need to be aware of, and assessing, the dimensions of difficulty as part of our solution. Am I missing some? Are you aware of any good guides?

Engaging Learning and the Serious eLearning Manifesto

9 July 2019 by Clark Leave a Comment

Way back in ’05, my book on games for learning was published. At its core was an alignment between what made an effective education practice and what makes engaging experiences. There were nine elements that characterized why learning should be ‘hard fun’. More recently, we released the Serious eLearning Manifesto. Here we had eight values that differentiated between ordinary elearning and serious elearning. So, the open question is how do these two lists match up? What is the alignment between Engaging Learning and the Serious eLearning manifesto?

The elements of the Serious eLearning Manifesto (SeM) are pretty straightforward. They’re listed as:

performance focused
meaningful to learners
engagement driven
authentic contexts
realistic decisions
real-world consequences
spaced practice
individualized challenges

The alignment (EEA: Effectiveness-Engagement Alignment) I found in Engaging Learning was based upon research I did on designing games for learning. I found elements that were repeated across proposals for effective education practice, and ones that were stipulated for engaging experiences. And I found a perfect overlap. Looking for a resolution between the two lists of elements looks something like:

clear goals
balanced challenge
context for the action
meaningful to domain
meaningful to learner
choice
active
consequences
novelty

And, with a little wordsmithing, I think we find a pretty good overlap! Obviously, not perfect, because they have different goals, but the important elements of a compelling learning experience emerge.

I could fiddle and suggest that clear goals are aligned to a performance focus, but instead that’s coming from making their learning be meaningful to the domain. I suggest that what really matters to organizations will be the ability to do, not know. So, really, the goals are implicit in the SeM; you shouldn’t be designing learning unless you have some learning goals!

Then, the balanced challenge is similar to the individualized challenge from the SeM. And context maps directly as well. As do consequences. And meaningfulness to learners. All these directly correspond.

Going a little further, I suggest that having choice (or appearance thereof) is important for realistic decisions. There should be alternatives that represent misconceptions about how to act. And, I suggest that the active focus is part of being engaging. Though, so too could novelty be. I’m not looking at multiple mappings but they would make sense as several things would combine to make a performance focus, as well as realistic decisions.

Other than that, on the EEA side the notion of novelty is more for engaging experiences than necessarily specific to serious elearning. On the SeM side, spaced practice is unique to learning. The notion of a game implies the ability for successful practice, so it’s implicit.

My short take, through this exercise, is to feel confident in both recommendations. We’re talking learning experience design here, and having the learning combine engagement as well is a nice outcome. I note that I’ll be running a Learning Experience Design workshop at DevLearn in October in Las Vegas, where’ll we’ll put these ideas to work. Hope to see you there!

The ITA Jay Cross Memorial Award for 2019: Michelle Ockers

5 July 2019 by Clark 1 Comment

Over a decade ago, my friend Jay Cross invited me to join the Internet Time Alliance. He had been touting the value of Informal Learning, and realized he was doing it alone. I was honored to join Jane Hart, Harold Jarche, and Charles Jennings, and have come to know and value them as colleagues and friends. When Jay passed away, we determined to honor his ideas by recognizing those who continue to carry the banner for informal learning. We announce the ITA memorial award on 5 July, Jay’s birthday. This year’s winner is Michelle Ockers.

I’ve only met Michelle once, when I was visiting Australia to deliver a keynote. She was kind enough to ask me to sign a copy of Revolutionize L&D. I didn’t know much about her work then, but have subsequently seen it in a variety of places. She’s active in social media, for instance. She also coordinated the Learning & Performance Institute capability map exercise that occurred last summer. She’s systematically demonstrated broad ranging interests and abilities around organizational learning.

I’m pleased that we can honor her and her work helping organizations work more productively and fluidly. The official announcement is on the ITA site. Congratulations, Michelle on the 2019 ITA Jay Cross Memorial Award!

Reconciling Cognitions and Contexts

3 July 2019 by Clark Leave a Comment

In my past two posts, I first looked at cognitions (situated, distributed, social) by contexts (think, work, and learn), and then the reverse. And, having filled out the matrixes anew, they weren’t quite the same. And that, I think, is the benefit of the exercise, a chance to think anew. So what emerged? Here’s the result of reconciling cognitions and contexts.

So, taking each cell back in the original pass of cognitions by contexts, what results? I took the Think row to, indeed, be Harold Jarche’s Seek > Sense > Share model (ok, my interpretation). We have in Situated, the feeds you’ve set up to see, and then the particular searches you need in the current context. Then, of course, you experiment and represent as ways to externalize thinking for Distributed. Finally, you share Socially.

For Work, not practices but principles (and the associated practices therefrom) as well as facilitation to support Situated Work. Performance support is, indeed, the Distributed support for Work. And Socially, you need to collaborate on specific tasks and cooperate in general.

Finally, for Learning, for a Situated world you need (spread) contextualized practice to support appropriate abstraction of the principles. You want models and examples to support performance in the practice, as Distributed resources. And, finally, for Social Learning, you need to communicate (e.g. discussions) and collaborate (group projects).

What’s changed is that I added search and feeds, and moved experiment, in the Think row. I went to principles from practices to support performance in ambiguity, left performance support untouched, and stayed with collaborations and cooperation instead of just shared representations (they’re part of collaborate). And, finally, I made practice about contexts, went from blended learning to support materials for learning, and interpreted social assignments as communicating and collaborating.

The question is, what does this mean? Does it give us any traction? I’m thinking it does, as it shifts the focus in what we’re doing to support folks. So I think it was interesting and valuable (to my thinking, at least ;) to consider reconciling cognitions and contexts.

Contexts By Cognitions

2 July 2019 by Clark Leave a Comment

So, in my last post, I talked about exploring the links between cognitions on the one hand (situated, distributed, social), and contexts (aligning with how we think, work, & learn). I did it one way, but then I thought to do it another, to instead consider Contexts by Cognitions, to see if I came to the same elements. And they weren’t quite identical! So I thought I should share that thinking, and then come to a reconciliation. Thinking out loud, as it were.

So in this one, I swapped the headings, emptied the matrix, and took a second stab at filling them out, with a relatively clear mind. (I generated the first diagram several days ago and had been iterating on it, but not today. Today I was writing it up and was early in the process, so I came to it relatively free of contamination. And of course, not completely, but this is ‘business significance’, not ‘statistical significance’ ;). The resulting diagram appears similar, but also some differences.

When we consider Thinking by Situated, we’re talking about coping with emergent situations. I thought being guided by best principles would be the way to cope, abstracted models. I thought representation was key for distributing one’s thinking, and sharing of course for social.

Working Situatedly suggested having in-house practices and facilitation. Of course, Distributed support for Work is performance support. And working socially suggests shared representations.

Finally, learning situated suggests the need for much practice (across contexts, I now think). Distributed support for learning are models and examples. And social learning suggests communicating (e.g. discussions) and collaboration (group projects).

Interestingly, these results differ from my previous post. So, I think I’ll have to reconcile them. The fact that I did get different results, and it sparked some additional thinking, is good. The outcome of considering contexts by cognitions improved the outcomes, I think. And that’s worth thinking about!