Archives for October 2023

How do you know what’s dubious research? There are lots of signals, more than I can cover in one post. However, a recent discovery serves as an example to illustrate some useful signals. I was trying to recall a paper I recently read, which suggested that reading is better than video for comprehending issues. Whether that’s true or not isn’t the issue. What is the issue is that in my search, I came across an article that really violated a number of principles. As I am wont to do, let’s briefly talk about bad research.

The title of the article (paraphrasing) was “Research confirms that video is superior to text”. Sure, that could be the case! (Actually the results say, not surprisingly, that one media’s better for some things, and another’s better at other; BTW, one of our great translators of research to practice, Patti Shank, has a series of articles on video that’s worth paying attention to.) Still, this article claimed to have a definitive statement about at least one study. However, when I looked at it, there were several problems.

First, the study was a survey asking instructors what they thought of video. That’s not the same as an experimental study! A good study would choose some appropriate content, and then have equivalent versions in text and video, and then have a comprehension test. (BTW, these experiments have been done.) Asking opinions, even of experts, isn’t quite as good. And these weren’t experts, they were just a collection of instructors. They might have valid opinions, but their expertise wasn’t a basis for deciding.

Worse, the folks conducting the study had. a. video. platform.  Sorry, that’s not an unbiased observer. They have a vested interest in the outcome. What we want is an impartial evaluation. This simply couldn’t be it. Not least, the author was the CEO of the platform.

It got worse. There was also a citation of the unjustified claim that images are processed 60K times better than text, yet the source of that claim hasn’t been found! They also cited learning styles! Citing unjustified claims isn’t a good practice in sound research. (For instance, when reviewing articles, I used to recommend rejecting them if they talked learning styles.) Yes, it wasn’t a research article on it’s own, but…I think misleading folks isn’t justified in any article (unless it’s illustrative and you then correct the situation).

Look, you can find valuable insights in lots of unexpected places, and in lots of unexpected ways. (I talk about ‘business significance’ can be as useful as statistical significance.) However, an author with a vested interest, using an inappropriate method, to make claims that are supported by debunked data, isn’t it. Please, be careful out there!

I’m currently advising several organizations on their approaches to the use of technology to support learning. Moreover, I’ve been doing so for more than two decades, and see a lot more such situations as well. One of the things that I struggle with is seeing folks getting all agog over new technology, yet without getting the design right beforehand.  Thus, let me make a simple suggestion: get the basics right first!

So, we know what leads to good learning. Heck, I’ve written a book summarizing what’s known about it, and I’m not the only one. Despite that fact that humans are complex, and increasingly so are our learning goals, there exist robust principles. We know that we should provide a sufficient quantity of appropriately challenging contextualized practice with aligned feedback, for instance. That is, if we actually want to achieve an outcome.

Yet, too often, we don’t see this. We see, instead, information presentation. Sometimes even with a knowledge test! Yet, such an effort is unlikely to lead to any meaningful change. That is, the investment’s wasted!

Worse, too often we see this being done with fancy new tools. Sure, I get as attracted to shiny new objects as anyone. However, I want to understand their core affordances for learning. Anyone had the dubious pleasure of attending a slide presentation in a virtual world? Or maybe being presented with animated presentations of lots of facts? The new tools may have a short-term effect of novelty, but that’s it. The fundamental aspects of how our brains learn are what’s going to make, or break, a learning investment.

On the other hand, if we start with getting the learning right, first, then there may be additional value coming from the tech. Adaptivity, on top of quality learning design, can accelerate the outcomes.  Immersion, at the right time and place, is better than not. Language models, properly used, can have big impacts. However, it comes from knowing the specific capabilities, and matching them to the need.

While I haven’t done the ‘back of the envelope’ calculation (I’m not a financial whiz), I can state with a fair degree of comfort that you’re better off doing simple learning with good design. Bad design with shiny tech is still bad design! You’ll more likely have an impact putting your investment into learning quality than using fancy tech to deliver dreck. Of course, once you’ve done that, the investment in tech can do a lot more!

I’m not against new tech, heck I’ve written on games, mobile, and more! What I’m against is new tech in lieu of good design. And I’m even more enamored of good tech on top of good design.  So, get the basics right first, then add in the shiny objects. That way you’re going to have a good return on your $$, and that’s a good thing. Right?

PS, speaking of basics, we’ll be running a debate tomorrow (11 Oct) discussing the Learning Experience Design (LXD) label. I’m sure we’ll unpack critical issues. Check it out. 

I was looking to wrap up my slide deck for DevLearn (will be talking learning science), and wanted to see if there was anything missing. In particular, I knew I had a fair bit on retention, but I was checking on what I call ‘transfer’. In doing so, I realized that colleagues and others use the term differently! Which means it’s time to talk ‘transfer’.

So one of our good research translators is Will Thalheimer. I found his report (PDF) on transfer, and its usage was as many others I’d found. Will describes transfer as moving learning from the instructional environment to the workplace. No argument this is important! Yet, it’s not how I use the word.

In talking learning science, I’ve seen and said that the two gold standards are retention over time until needed, and transfer to all appropriate (and no inappropriate) situations. (That’s what I’ll be talking about in my session: what leads to those.) My notions here are both part of his transfer.

I get my usage of transfer essentially from folks like John Anderson at Carnegie Mellon Uni, who talks about what facilitates transfer. He’s the one I think of, for instance, when thinking of identical elements transfer, which stipulates that the more elements are the same, the higher the transfer. There’s near and far transfer, as well. That indicates the breadth of area you’re transferring to, really. E.g. negotiation would be far transfer, as there are many situations, whereas training to specifically run projector MX3600D4 is very much near transfer.

In a review of transfer I found (PDF), a distinction is made between ‘low-‘ and ‘high-road’ transfer. The former ‘happens’, while the latter is engineered. Given that, to me, instruction is about engineering success, maximizing likelihoods, I’m all for the latter. So, choosing contexts and supporting reflection.

To memory, even the academic literature about workplace learning seems to support the other interpretation. Yet, my term seems more in line with what the learning science community. In the longer term, we’ll need to reconcile, but for me, the distinction between the elements of retention and transfer are important. So, when I’m talking ‘transfer’, what I mean is the breadth of application. That’s my take, what are your thoughts?