Search Results for: Reading Research

Reading Research?

14 September 2021 by Clark Leave a Comment

I was honored to have a colleague laud my Myths book (she was kind enough to also promote the newer learning science book), but it was something she said that I found intriguing. She suggested that one of the things in it includes “discussing how to read research”. And it occurs to me that it’s worth unpacking the situation a wee bit more. So here’s a discussion about how we (properly) develop learning science that informs us in reading research.

Caveat: I haven’t been an active researcher for decades, serving instead to interpret and apply the research, but it’s easier to say ‘we’ than “scientists”, etc.

Generally, theory drives research. You’ve created an explanation that accounts for observed phenomena better than previous approaches. What you do then is extend it to other predictions, and test them. Occasionally, we do purely exploratory studies just to see what emerges, but mostly we generate hypotheses and test them.

We do this with some rigor. We try to ensure that the method we devise removes confounding variables, and then we use statistical analysis to remove the effects of other factors. For instance, I created a convoluted balancing approach to remove order effects in my Ph.D. research. (So complicated that I had to analyze a factor or two first, to ensure it wasn’t a factor, so I could remove it from the resulting analysis!). We also try to select relevant subjects, design uncontaminated materials, and carefully control our analysis. Understanding the ways in which we do this requires an ability to know about experiment design, which isn’t common knowledge.

Moreover, we then need to share this with our colleagues so that they can review what we’ve done. We need to do it in unambiguous language, using the specific vocabulary of our field. And we need to make it scrutable. Thus, we publish in peer-reviewed journals which mean others have looked at our work and deemed it acceptable. However, the language is deliberately passive, unemotional, and precise, as well as focused on a very narrow topic. Thus, it’s not a lot of fun to read unless you really care about the topic!

There are problems with this. Increasingly, we’re finding that trying to isolate independent variables doesn’t reflect the inherent interactions. Our brains actually have a lot of complexity that hinder simple explanations. We’ve also found that it’s difficult to get representative subjects, when what’s easy to get are higher education students in the developed world. There are also politics involved, sad to say, so that it can be hard for new ideas to emerge if they challenge the entrenched views. Yet, it’s still the best approach we have. The scientific method has led to more advances in understanding than anything else!

There are things to worry about as a consumer of science. For one, there are people who fake results. They’re few, of course. There’s also research that’s kept proprietary, for financial reasons. Or is commissioned. As soon as there’s money involved, there’s the opportunity for corruption (think: tobacco, and sugar). Companies may have something that they tout as valid, but the research base isn’t publically available. Caveat emptor!

Thus, being able to successfully read research isn’t for everyone. You need to be able to comprehend the studies, and know when to be wary. The easy thing to do is to look for translations, and translators, who have demonstrated a trustworthy ability to help sort out the wheat from the chaff. They exist.

I hope this illustrates what reading research requires. You can take some preliminary steps: give it the ‘sniff’ test, see if it applies to you, and see who’s telling you this (and if anyone else is agreeing or saying to the contrary) and what their stake in the game is. If these steps don’t answer a question, however, maybe you want to look for good guidance. Make sense?

Bad research

17 October 2023 by Clark 1 Comment

How do you know what’s dubious research? There are lots of signals, more than I can cover in one post. However, a recent discovery serves as an example to illustrate some useful signals. I was trying to recall a paper I recently read, which suggested that reading is better than video for comprehending issues. Whether that’s true or not isn’t the issue. What is the issue is that in my search, I came across an article that really violated a number of principles. As I am wont to do, let’s briefly talk about bad research.

The title of the article (paraphrasing) was “Research confirms that video is superior to text”. Sure, that could be the case! (Actually the results say, not surprisingly, that one media’s better for some things, and another’s better at other; BTW, one of our great translators of research to practice, Patti Shank, has a series of articles on video that’s worth paying attention to.) Still, this article claimed to have a definitive statement about at least one study. However, when I looked at it, there were several problems.

First, the study was a survey asking instructors what they thought of video. That’s not the same as an experimental study! A good study would choose some appropriate content, and then have equivalent versions in text and video, and then have a comprehension test. (BTW, these experiments have been done.) Asking opinions, even of experts, isn’t quite as good. And these weren’t experts, they were just a collection of instructors. They might have valid opinions, but their expertise wasn’t a basis for deciding.

Worse, the folks conducting the study had. a. video. platform. Sorry, that’s not an unbiased observer. They have a vested interest in the outcome. What we want is an impartial evaluation. This simply couldn’t be it. Not least, the author was the CEO of the platform.

It got worse. There was also a citation of the unjustified claim that images are processed 60K times better than text, yet the source of that claim hasn’t been found! They also cited learning styles! Citing unjustified claims isn’t a good practice in sound research. (For instance, when reviewing articles, I used to recommend rejecting them if they talked learning styles.) Yes, it wasn’t a research article on it’s own, but…I think misleading folks isn’t justified in any article (unless it’s illustrative and you then correct the situation).

Look, you can find valuable insights in lots of unexpected places, and in lots of unexpected ways. (I talk about ‘business significance’ can be as useful as statistical significance.) However, an author with a vested interest, using an inappropriate method, to make claims that are supported by debunked data, isn’t it. Please, be careful out there!

New recommended readings

8 June 2021 by Clark Leave a Comment

Of late, I‘ve been reading quite a lot, and I‘m finding some very interesting books. Not all have immediate take homes, but I want to introduce a few to you with some notes. Not all will be relevant, but all are interesting and even important. I‘ll also update my list of recommended readings. So here are my new recommended readings. (With Amazon Associates links: support your friendly neighborhood consultants.)

First, of course, I have to point out my own Learning Science for Instructional Designers. A self-serving pitch confounded with an overload of self-importance? Let me explain. I am perhaps overly confident that it does what it says, but others have said nice things. I really did design it to be the absolute minimum reading that you need to have a scrutable foundation for your choices. Whether it succeeds is an open question, so check out some of what others are saying. As to self-serving, unless you write an absolute mass best-seller, the money you make off books is trivial. In my experience, you make more money giving it away to potential clients as a better business card than you do on sales. The typically few hundred dollars I get a year for each book aren‘t going to solve my financial woes! Instead, it‘s just part of my campaign to improve our practices.

So, the first book I want to recommend is Annie Murphy Paul‘s The Extended Mind. She writes about new facets of cognition that open up a whole area for our understanding. Written by a journalist, it is compelling reading. Backed in science, it’s valuable as well. In the areas I know and have talked about, e.g. emergent and distributed cognition, she gets it right, which leads me to believe the rest is similarly spot on. (Also her previous track record; I mind-mapped her talk on learning myths at a Learning Solutions conference). Well-illustrated with examples and research, she covers embodied cognition, situated cognition, and socially distributed cognition, all important. Moreover, there‘re solid implications for the redesign of instruction. I‘ll be writing a full review later, but here‘s an initial recommendation on an important and interesting read.

I‘ll also alert you to Tania Luna‘s and LeeAnn Renninger‘s Surprise. This is an interesting and fun book that instead of focusing on learning effectiveness, looks at the engagement side. As their subtitle suggests, it‘s about how to Embrace the Unpredictable and Engineer the Unexpected. While the first bit of that is useful personally, it‘s the latter that provides lots of guidance about how to take our learning from events to experiences. Using solid research on what makes experiences memorable (hint: surprise!) and illustrative anecdotes, they point out systematic steps that can be used to improve outcomes. It‘s going to affect my Make It Meaningful work!

Then, without too many direct implications, but intrinsically interesting is Lisa Feldman Barrett‘s How Emotions Are Made. Recommended to me, this book is more for the cog sci groupie, but it does a couple of interesting things. First, it creates a more detailed yet still accessible explanation of the implications of Karl Friston‘s Free Energy Theory. Barrett talks about how those predictions are working constantly and at many levels in a way that provides some insights. Second, she then uses that framework to debunk the existing models of emotions. The experiments with people recognizing facial expressions of emotion get explained in a way that makes clear that emotions are not the fundamental elements we think they are. Instead, emotions social constructs! Which undermines, BTW, all the facial recognition of emotion work.

I also was pointed to Tim Harford‘s The Data Detective, and I do think it‘s a well done work about how to interpret statistical claims. It didn‘t grip me quite as viscerally as the afore-mentioned books, but I think that‘s because I (over-)trust my background in data and statistics. It is a really well done read about some simple but useful rules for how to be a more careful reviewer of statistical claims. While focused on parsing the broader picture of societal claims (and social media hype), it is relevant to evaluating learning science as well.

I hope you find my new recommended readings of interest and value. Now, what are you recommending to me? (He says, with great trepidation. ;)

Theory or Research?

17 July 2019 by Clark Leave a Comment

There’s a lot of call for evidence-based methods (as mentioned yesterday): L&D, learning design, and more. And this is a good thing. But…do you want to be basing your steps on a particular empirical study, or the framework within which that study emerged? Let me make the case for one approach. My answer to theory or research is theory. Here’s why.

Most research experiments are done in the context of a theoretical framework. For instance, the work on worked examples comes from John Sweller’s Cognitive Load theory. Ann Brown & Ann-Marie Palincsar’s experiments on reading were framed within Reciprocal Teaching, etc. Theory generates experiments which refine theory.

The individual experiments illuminate aspects of the broader perspective. Researchers tend to run experiments driven by a theory. The theory leads to a hypothesis, and then that hypothesis is testable. There are some exploratory studies done, but typically a theoretical explanation is generated to explain the results. That explanation is then subject to further testing.

Some theories are even meta-theories! Collins & Brown’s Cognitive Apprenticeship (a favorite) is based upon integrating several different theories, including the Reciprocal Teaching, Alan Schoenfeld’s work on examples in math, and the work of Scardemalia & Bereiter on scaffolding writing. And, of course, most theories have to account for others’ results from other frameworks if they’re empirically sound.

The approach I discuss in things like my Learning Experience Design workshops is a synthesis of theories as well. It’s an eclectic mix including the above mentioned, Cognitive Flexibility, Elaboration, ARCS, and more. If I were in a research setting, I’d be conducting experiments on engagement (pushing beyond ARCS) to test my own theories of what makes experiences as engaging and effective. Which, not coincidentally, was the research I was doing when I was an academic (and led to Engaging Learning). (As well as integration of systems for a ubiquitous coaching environment, which generates many related topics.)

While individual results, such as the benefits of relearning, are valuable and easy to point to, it’s the extended body of work on topics that provides for longevity and applicability. Any one study may or may not be directly applicable to your work, but the theoretical implications give you a basis to make decisions even in situations that don’t directly map. There’s the possibility to extend to far, but it’s better than having no guidance at all.

Having theories to hand that complement each other is a principled way to design individual solutions and design processes. Similarly for strategic work as well (Revolutionize L&D) is a similar integration of diverse elements to make a coherent whole. Knowing, and mastering, the valid and useful theories is a good basis for making organizational learning decisions. And avoiding myths! Being able to apply them, of course, is also critical ;).

So, while they’re complementary, in the choice between theory or research I’ll point to one having more utility. Here’s to theories and those who develop and advance them!

Deeper Learning Reading List

20 April 2016 by Clark 3 Comments

So, for my last post, I had the Revolution Reading List, and it occurred to me that I’ve been reading a bit about deeper learning design, too, so I thought I’d offer some pointers here too.

The starting point would be Julie Dirksen’s Design For How People Learn (already in it’s 2nd edition). It’s a very good interpretation of learning research applied to design, and very readable.

A new book that’s very good is Make It Stick, by Peter Brown, Henry Roediger III, and Mark McDaniel, the former being a writer who’s worked with two scientists to take learning research into 10 principles.

And let me mention two Ruth Clark books. One with Dick Mayer from UCSB, e-Learning and the Science of Instruction, that focuses on the use of media. A second with Frank Nguyen and the wise John Sweller, Efficiency in Learning, focuses on cognitive load (which has many implications, including some overlap with the first).

Patti Schank has come out with a concise compilation of research called The Science of Learning that’s available to ATD members. Short and focused with her usual rigor. If you’re not an ATD member, you can read her blog posts that contributed (click ‘View All’).

Dorian Peters book on Interface Design for Learning also has some good learning principles as well as interface design guidance. It’s not the same for learning as for doing.

Of course, a classic is a compilation of research by a blue-ribbon team lead by John Bransford, How People Learn, (online or downloadable). Voluminous, but pretty much state of the art.

Another classic is the Cognitive Apprenticeship model of Allen Collins & John Seely Brown. A holistic model abstracted across some seminal work, and quite readable.

The Science of Learning Center has an academic integration of research to instruction theory by Ken Koedinger, et al, The Knowledge-Learning-Instruction Framework, that’s freely available as a PDF.

I’d be remiss if I don’t point out the Serious eLearning Manifesto, which has 22 research principles underneath the 8 values that differentiate serious elearning from typical versions. If you buy in, please sign on!

And, of course, I can point you to my own series for Learnnovators on Deeper ID.

So there you go with some good material to get you going. We need to do better at elearning, treating it with the importance it deserves. These don’t necessarily tell you how to redevelop your learning design processes, but you know who can help you with that. What’s on your list?

Design Readings

31 May 2012 by Clark 4 Comments

Another book on design crossed my radar when I was at a retreat and in the stack of one of the other guests was Julie Dirksen’s book Design for How People Learn and Susan Weinschenk’s 100 Things Every Designer Needs to Know About People. This book provides a nice complement to Julie’s, focusing on straight facts about how we process the world.

Dr. Weinschenk’s book systematically goes through categories of important design considerations:

How People See
How People Read
How People Remember
How People Think
How People Focus Their Attention
What Motivates People
People Are Social Animals
How People Feel
People Make Mistakes
How People Decide

Under each category are important points, described, buttressed by research, and boiled down into useful guidelines. This includes much of the research I talk about when I discuss deeper Instructional Design, and more. While it’s written for UI designers mostly, it’s extremely relevant to learning design as well. And it’s easy reading and reference, illustrated and to-the-point.

There are some really definitive books that people who design for people need to have read or have to hand. This fits into the latter category as does Dirksen’s book, while Don Norman’s books, e.g. Design of Everyday Things fit into the former. Must knows and must haves.

Learning culture = design culture?

21 April 2026 by Clark Leave a Comment

I”m reading Don Norman’s Design for a Better World (recommended; more after I finish). One of the things mentioned is ‘design culture’. Now, I’ve been a big fan of ‘learning culture’, and so was triggered. What’s the relationship? Some thoughts.

So, to start, I think of learning culture as a determinant of performance ecosystem success. I’ve regularly touted Garvin, Edmondson, & Gino’s model as a grounded approach that identifies major factors. The issue is creating an environment where people contribute their best ideas, and folks are learning ‘together’. Practices such as Jane Bozarth’s Show Your Work are components, as is Amy Edmonson’s The Fearless Organization about psychological safety.

So what is a ‘design culture’? Don doesn’t define it, but Wikipedia says “approaches that improve customer experiences through design.” Of course, the customers may be internal, I’d suggest. More importantly, it’s about design being a core component of the way things are done. Yet, isn’t design ‘learning’?

I’d suggest that, indeed, design is learning. I’ve suggested in the past (e.g. here) that design, research, and trouble-shooting are informal learning, as you don’t know the answer when you start. Design is creating a solution where one doesn’t exist. It’s part of learning: you design, test and measure, and iterate until the metrics meet your needs. Thus, I’d argue that for a successful design culture, you’d need a successful learning culture, or you can’t get the best solutions.

The bigger argument of Don’s book is about how things have gone south, societally/globally, and the role of design in causing and remedying that. I’m sympathetic, I admit. I do feel chuffed that he’s mentioning participatory design, something I just talked about for the UX part of the Learning & Development as Ecosystem conference. But his overall message is important, particularly in light of his previous works on design, notably Design of Everyday Things. If he’s now saying he got it wrong, I’ll suggest we probably should take note of what he’s now thinking.

So, I might take design culture as a subset of a learning culture, or an integral part. However, if SDT (c.f. Matt Richter’s forthcoming The Motivation Blueprint, and Stephen Johnson’s previous Drive) is right, considering our purpose may need to go beyond not just doing harm, and look at remedying what’s wrong. My quibble about culture is relatively small potatoes in the bigger picture. Still, worth pondering.

Key notes

16 December 2025 by Clark Leave a Comment

I’ve seen a lot of keynotes over the years. I’ve even given them! It’s time to reconcile my thoughts. So here are some key notes on keynotes.

One of the things that I’ve seen is flawless performances. Now, one of the things you’re told is that the focus is on you, and slides only should be used as an augment, not on all the time. I confess I’m not good about that (I am not very comfortable in the spotlight; imposter syndrome I suppose). Another is that you should have pauses, and jokes, and such. I do a pretty good dramatic reading (I won the dirty limerick reading contest at work once!), and occasionally even manage to raise a smile or two. The best, however, have their patter completely down. I can’t do that, because my thoughts are continually evolving, but I admire it when it happens. And I’m pretty good at tailoring a talk to the audience (having learned a few times the hard way!).

Inspiration is good, too. Letting people know there’s a way to surpass this barrier works! I try to do that too, though I confess I talk about learning design, not achieving things like climbing Mt Everest (really heard a keynote about that, and it was cool!). And I probably am a bit too conceptual, though I am learning to do better about grounding my principles in practice. However, I do recoil from too much ‘enthusiasm’! Somehow it comes across as artificial. But then, I can be a bit of a curmudgeon (apparently)…

However, what really matters to me is accuracy. I don’t mind if folks are a bit enthusiastic or polished, but I really get wound around the axle a bit when folks state stuff that’s just wrong. For instance, I heard a well-regarded personage opine about games, something I know a wee bit about (my first job, back when dinosaurs strode the earth, was on games, and it’s been a recurrent them in practice, research, and writing for literally decades). And that individual said something just dead wrong. As you may surmise, it really ground my gears. Similarly with learning science, or mobile. In general, when people have beautifully symmetric ‘n part models’ without grounding, I want to know if those are convenient, or a necessary and sufficient list. (Too often the former.)

I also like when people tout doing things I believe in, but when it’s an area I know about, you better agree with the science, and present it accurately. If you don’t, well, I won’t be quiet about it. (I guess it’s a flaw in my character!) Still, when you say these are the five things to X, and they’re a) not completely separable, b) incomplete, c) wrong, etc, I’m going to be turned off.

Look, I love a good keynote. Many times, they get people who aren’t from the field where the keynote’s presented, and they make connections from adjacent fields. That’s acceptable, even desirable! I like a well-presented talk as well as the next person. I like ideas, and even inspiration. But I will complain about bad information. Always. Those are my key notes on keynotes, what are yours? (And I’m available, if you want L&D advice ahead of the curve but grounded in evidence. Particularly contrary takes… ;)

Conference season

28 October 2025 by Clark Leave a Comment

Conference season has commenced. Two are already in the books. Three I know about are coming up, and I’m playing a role in two. So, what’s up, and when? Here’s what I know.

So, first, the Learning Development Accelerator (LDA) is running the Creating a Motivating Work Environment Summit. This is in conjunction with the Center for Self-Determination Theory, so it’s scrutable. I’m not part of this, except as a participant. It follows the usual LDA format: access to videos created by the presenters, followed by live sessions at two different times. The videos are already up, and the live sessions are coming soon, Nov 3 – 7! It’s all online, which makes it easy to attend, and the live sessions will be recorded. There’s a stellar lineup of speakers, naturally! I’m increasingly finding the value in the theory, so I look forward to the session. Caveat: I’m a Co-Director of the LDA, so I have a vested interest in the success, but I still think it’s of interest (at least to me).

Then, the Learning Guild is holding the next DevLearn conference, and I’ll be doing several things. My Wed is pretty full, as I’m starting by hosting a Morning Buzz on building a learning culture. Hosting isn’t the same as presenting, but instead just facilitating the conversation. Then I’m presenting on the spacing of learning. I’ve been actively engaged in developing a spaced learning strategy, and will be sharing the key principles from learning science research. As well as what’s not (yet) known! I’ll be signing books right after that at the event bookstore. On Thursday, I’m part of a panel on AI (which will probably be interpreted as Generative AI), and will be my usual critical self ;). Of course, I’ll also be wandering the halls and expo. If you’re there, say hello!

Finally, the LDA is also running our second Learning Science Conference. (If you attended last year, you get a big discount!). It uses the same format asa the Motivation Summit, above, that is with specifically curated content in presentations, and live sessions Dec 8 – 12. It starts 3 Nov, and I’m active in this one too. I’ll again be presenting two sections. The first will be on getting information into, and out of, long-term memory, specifically generative and retrieval practice. I talked about the latter, last year, so I’m refining that (my own understanding evolves as does the field), and adding more on generative. Similarly with social and informal learning, which I’ll be presenting again. That is, I’ll be rehashing the old, and adding a bit new.

There’re new speakers, too. We’ve the honor of having Gale Sinatra and Jim Hewitt, and Rich Mayer will be doing a special session with Ruth Clark. Other presenters include my fellow co-Director, Matt Richter, along with Stella Lee and Nidhi Sachdeva. There’ll be special sessions, such as with Will Thalheimer. Of course, we’ll have a debate, here with me going head to head with authors Bianca Baumann and Mike Taylor on marketing and motivation. We also will have a panel with greats Julie Dirksen, Jane Bozarth, and Koreen Pagano. And more.

Sure, there’re lots of ways to get on top of learning science and good design. There’s the Serious eLearning Manifesto, books (e.g. my recommended reading list), blogs (like this one), magazines likeTraining, eLearn, journals, and more. However, getting together with your fellow practitioners, live or online, is a real boon, and so conference season is a great opportunity. Hope to see you somewhere soon!

Transforming from knowledge to performance

16 September 2025 by Clark Leave a Comment

As I’ve mentioned, I’m working with a startup looking at extending training through small LIFTs. The problem is that most training is ‘event’ based, where learning is in a concentrated time. Which is fine for performing right after. However, much of what we train for are things that may or may not happen soon. What we want is to go from the knowledge after the event to actually performing in new ways after the event, possibly a long time. We need retention from the learning to the situation, and transfer to all appropriate (and no inappropriate) situations. Thus, we need to think differently. And, as I suggested, we’re looking at supporting people not just with formal learning, but beyond, to developing their ability over time. We really want to be transforming from knowledge to performance. So, what’s that look like?

As usual, when I’m supposed to be sleeping is one of the times I end up noodling things over. And, so it was some nights ago. I was thinking about (as I’m wont to do) the cognitive roles that we need. I talk about practice, and models, and examples, and more recently, generative activities. But that’s formal learning, and we have a good evidence base for that. But what about going forward? What sorts of activities make sense?

Here I’m going out of my comfort zone. Yes, I’ve been doing some reading about coaching, particularly domain-independent vs domain-specific coaching. Now, here I don’t necessarily know what the research says specifically, but I do see the convergence of a variety of different models. So, I can make inferences. And post them here to get corrected!

As you might expect, I made a diagram to help me understand. So, I reckon there’s an early, mid, and late stage of development of capability. Formal learning should really be about getting you ready to apply.

That is the early phase which includes reflection (really, a generative activity), which can be personal (ala scripts) or conceptual (schemas). Also, reactivation. That is, seeing different ways of looking at it (new models), more examples in context, and of course more practice. (Retrieval practice, of course, where you’re applying the knowledge.)

Then, in mid-phase, your learners are applying, but to real situations, not simulations. Their initial plan on how to apply the knowledge might be part of the end of the early stage, but then it’s time to apply. Which could (should?) lead to revisions of the plan, and on reflecting on any barriers. Those barriers could be internal (their own understanding or hangups), or external (lack of resources, situations, tools, etc). The former are grounds for discussion, the latter for action on the part of the org!

Then, at the late stage, learners should be looking at the impact. They can reflect on the impact on them, which could also be a mid-phase action, but ultimately you want to see if they’re having an impact overall. Then, of course, you could want to survey about the learning experience itself. While it’s all data, the org impact is useful data to evaluate what’s going on and how it’s going, and the survey can help you continue to improve either this or your next initiative.

Those’re my initial thoughts on transforming from knowledge to performance. There’s some overlap, no doubt, e.g. you could continue sending reapplications if there aren’t frequent opportunities in the real world. Likewise, your learners should be assessing impact in the need to revise a plan. Still, this seems to make sense in the first instance, at least to me. (Addressing the ‘when’, how much and what spacing, is what I’ll be talking about at DevLearn. ;) Now, it’s over to you. What have I got wrong, am missing, …?