Search Results for: Reading Research

Reading Research?

14 September 2021 by Clark Leave a Comment

I was honored to have a colleague laud my Myths book (she was kind enough to also promote the newer learning science book), but it was something she said that I found intriguing. She suggested that one of the things in it includes “discussing how to read research”. And it occurs to me that it’s worth unpacking the situation a wee bit more. So here’s a discussion about how we (properly) develop learning science that informs us in reading research.

Caveat: I haven’t been an active researcher for decades, serving instead to interpret and apply the research, but it’s easier to say ‘we’ than “scientists”, etc.

Generally, theory drives research. You’ve created an explanation that accounts for observed phenomena better than previous approaches. What you do then is extend it to other predictions, and test them. Occasionally, we do purely exploratory studies just to see what emerges, but mostly we generate hypotheses and test them.

We do this with some rigor. We try to ensure that the method we devise removes confounding variables, and then we use statistical analysis to remove the effects of other factors. For instance, I created a convoluted balancing approach to remove order effects in my Ph.D. research. (So complicated that I had to analyze a factor or two first, to ensure it wasn’t a factor, so I could remove it from the resulting analysis!). We also try to select relevant subjects, design uncontaminated materials, and carefully control our analysis. Understanding the ways in which we do this requires an ability to know about experiment design, which isn’t common knowledge.

Moreover, we then need to share this with our colleagues so that they can review what we’ve done. We need to do it in unambiguous language, using the specific vocabulary of our field. And we need to make it scrutable. Thus, we publish in peer-reviewed journals which mean others have looked at our work and deemed it acceptable. However, the language is deliberately passive, unemotional, and precise, as well as focused on a very narrow topic. Thus, it’s not a lot of fun to read unless you really care about the topic!

There are problems with this. Increasingly, we’re finding that trying to isolate independent variables doesn’t reflect the inherent interactions. Our brains actually have a lot of complexity that hinder simple explanations. We’ve also found that it’s difficult to get representative subjects, when what’s easy to get are higher education students in the developed world. There are also politics involved, sad to say, so that it can be hard for new ideas to emerge if they challenge the entrenched views. Yet, it’s still the best approach we have. The scientific method has led to more advances in understanding than anything else!

There are things to worry about as a consumer of science. For one, there are people who fake results. They’re few, of course. There’s also research that’s kept proprietary, for financial reasons. Or is commissioned. As soon as there’s money involved, there’s the opportunity for corruption (think: tobacco, and sugar). Companies may have something that they tout as valid, but the research base isn’t publically available. Caveat emptor!

Thus, being able to successfully read research isn’t for everyone. You need to be able to comprehend the studies, and know when to be wary. The easy thing to do is to look for translations, and translators, who have demonstrated a trustworthy ability to help sort out the wheat from the chaff. They exist.

I hope this illustrates what reading research requires. You can take some preliminary steps: give it the ‘sniff’ test, see if it applies to you, and see who’s telling you this (and if anyone else is agreeing or saying to the contrary) and what their stake in the game is. If these steps don’t answer a question, however, maybe you want to look for good guidance. Make sense?

New recommended readings

8 June 2021 by Clark Leave a Comment

Of late, I’ve been reading quite a lot, and I’m finding some very interesting books. Not all have immediate take homes, but I want to introduce a few to you with some notes. Not all will be relevant, but all are interesting and even important. I’ll also update my list of recommended readings. So here are my new recommended readings. (With Amazon Associates links: support your friendly neighborhood consultants.)

First, of course, I have to point out my own Learning Science for Instructional Designers. A self-serving pitch confounded with an overload of self-importance? Let me explain. I am perhaps overly confident that it does what it says, but others have said nice things. I really did design it to be the absolute minimum reading that you need to have a scrutable foundation for your choices. Whether it succeeds is an open question, so check out some of what others are saying. As to self-serving, unless you write an absolute mass best-seller, the money you make off books is trivial. In my experience, you make more money giving it away to potential clients as a better business card than you do on sales. The typically few hundred dollars I get a year for each book aren’t going to solve my financial woes! Instead, it’s just part of my campaign to improve our practices.

So, the first book I want to recommend is Annie Murphy Paul’s The Extended Mind. She writes about new facets of cognition that open up a whole area for our understanding. Written by a journalist, it is compelling reading. Backed in science, it’s valuable as well. In the areas I know and have talked about, e.g. emergent and distributed cognition, she gets it right, which leads me to believe the rest is similarly spot on. (Also her previous track record; I mind-mapped her talk on learning myths at a Learning Solutions conference). Well-illustrated with examples and research, she covers embodied cognition, situated cognition, and socially distributed cognition, all important. Moreover, there’re solid implications for the redesign of instruction. I’ll be writing a full review later, but here’s an initial recommendation on an important and interesting read.

I’ll also alert you to Tania Luna’s and LeeAnn Renninger’s Surprise. This is an interesting and fun book that instead of focusing on learning effectiveness, looks at the engagement side. As their subtitle suggests, it’s about how to Embrace the Unpredictable and Engineer the Unexpected. While the first bit of that is useful personally, it’s the latter that provides lots of guidance about how to take our learning from events to experiences. Using solid research on what makes experiences memorable (hint: surprise!) and illustrative anecdotes, they point out systematic steps that can be used to improve outcomes. It’s going to affect my Make It Meaningful work!

Then, without too many direct implications, but intrinsically interesting is Lisa Feldman Barrett’s How Emotions Are Made. Recommended to me, this book is more for the cog sci groupie, but it does a couple of interesting things. First, it creates a more detailed yet still accessible explanation of the implications of Karl Friston’s Free Energy Theory. Barrett talks about how those predictions are working constantly and at many levels in a way that provides some insights. Second, she then uses that framework to debunk the existing models of emotions. The experiments with people recognizing facial expressions of emotion get explained in a way that makes clear that emotions are not the fundamental elements we think they are. Instead, emotions social constructs! Which undermines, BTW, all the facial recognition of emotion work.

I also was pointed to Tim Harford’s The Data Detective, and I do think it’s a well done work about how to interpret statistical claims. It didn’t grip me quite as viscerally as the afore-mentioned books, but I think that’s because I (over-)trust my background in data and statistics. It is a really well done read about some simple but useful rules for how to be a more careful reviewer of statistical claims. While focused on parsing the broader picture of societal claims (and social media hype), it is relevant to evaluating learning science as well.

I hope you find my new recommended readings of interest and value. Now, what are you recommending to me? (He says, with great trepidation. ;)

Theory or Research?

17 July 2019 by Clark Leave a Comment

There’s a lot of call for evidence-based methods (as mentioned yesterday): L&D, learning design, and more. And this is a good thing. But…do you want to be basing your steps on a particular empirical study, or the framework within which that study emerged? Let me make the case for one approach. My answer to theory or research is theory. Here’s why.

Most research experiments are done in the context of a theoretical framework. For instance, the work on worked examples comes from John Sweller’s Cognitive Load theory. Ann Brown & Ann-Marie Palincsar’s experiments on reading were framed within Reciprocal Teaching, etc. Theory generates experiments which refine theory.

The individual experiments illuminate aspects of the broader perspective. Researchers tend to run experiments driven by a theory. The theory leads to a hypothesis, and then that hypothesis is testable. There are some exploratory studies done, but typically a theoretical explanation is generated to explain the results. That explanation is then subject to further testing.

Some theories are even meta-theories! Collins & Brown’s Cognitive Apprenticeship (a favorite) is based upon integrating several different theories, including the Reciprocal Teaching, Alan Schoenfeld’s work on examples in math, and the work of Scardemalia & Bereiter on scaffolding writing. And, of course, most theories have to account for others’ results from other frameworks if they’re empirically sound.

The approach I discuss in things like my Learning Experience Design workshops is a synthesis of theories as well. It’s an eclectic mix including the above mentioned, Cognitive Flexibility, Elaboration, ARCS, and more. If I were in a research setting, I’d be conducting experiments on engagement (pushing beyond ARCS) to test my own theories of what makes experiences as engaging and effective. Which, not coincidentally, was the research I was doing when I was an academic (and led to Engaging Learning). (As well as integration of systems for a ubiquitous coaching environment, which generates many related topics.)

While individual results, such as the benefits of relearning, are valuable and easy to point to, it’s the extended body of work on topics that provides for longevity and applicability. Any one study may or may not be directly applicable to your work, but the theoretical implications give you a basis to make decisions even in situations that don’t directly map. There’s the possibility to extend to far, but it’s better than having no guidance at all.

Having theories to hand that complement each other is a principled way to design individual solutions and design processes. Similarly for strategic work as well (Revolutionize L&D) is a similar integration of diverse elements to make a coherent whole. Knowing, and mastering, the valid and useful theories is a good basis for making organizational learning decisions. And avoiding myths! Being able to apply them, of course, is also critical ;).

So, while they’re complementary, in the choice between theory or research I’ll point to one having more utility. Here’s to theories and those who develop and advance them!

Deeper Learning Reading List

20 April 2016 by Clark 3 Comments

So, for my last post, I had the Revolution Reading List, and it occurred to me that I’ve been reading a bit about deeper learning design, too, so I thought I’d offer some pointers here too.

The starting point would be Julie Dirksen’s Design For How People Learn (already in it’s 2nd edition). It’s a very good interpretation of learning research applied to design, and very readable.

A new book that’s very good is Make It Stick, by Peter Brown, Henry Roediger III, and Mark McDaniel, the former being a writer who’s worked with two scientists to take learning research into 10 principles.

And let me mention two Ruth Clark books. One with Dick Mayer from UCSB, e-Learning and the Science of Instruction, that focuses on the use of media. A second with Frank Nguyen and the wise John Sweller, Efficiency in Learning, focuses on cognitive load (which has many implications, including some overlap with the first).

Patti Schank has come out with a concise compilation of research called The Science of Learning that’s available to ATD members. Short and focused with her usual rigor. If you’re not an ATD member, you can read her blog posts that contributed (click ‘View All’).

Dorian Peters book on Interface Design for Learning also has some good learning principles as well as interface design guidance. It’s not the same for learning as for doing.

Of course, a classic is a compilation of research by a blue-ribbon team lead by John Bransford, How People Learn, (online or downloadable). Voluminous, but pretty much state of the art.

Another classic is the Cognitive Apprenticeship model of Allen Collins & John Seely Brown. A holistic model abstracted across some seminal work, and quite readable.

The Science of Learning Center has an academic integration of research to instruction theory by Ken Koedinger, et al, The Knowledge-Learning-Instruction Framework, that’s freely available as a PDF.

I’d be remiss if I don’t point out the Serious eLearning Manifesto, which has 22 research principles underneath the 8 values that differentiate serious elearning from typical versions. If you buy in, please sign on!

And, of course, I can point you to my own series for Learnnovators on Deeper ID.

So there you go with some good material to get you going. We need to do better at elearning, treating it with the importance it deserves. These don’t necessarily tell you how to redevelop your learning design processes, but you know who can help you with that. What’s on your list?

Design Readings

31 May 2012 by Clark 4 Comments

Another book on design crossed my radar when I was at a retreat and in the stack of one of the other guests was Julie Dirksen’s book Design for How People Learn and Susan Weinschenk’s 100 Things Every Designer Needs to Know About People. This book provides a nice complement to Julie’s, focusing on straight facts about how we process the world.

Dr. Weinschenk’s book systematically goes through categories of important design considerations:

How People See
How People Read
How People Remember
How People Think
How People Focus Their Attention
What Motivates People
People Are Social Animals
How People Feel
People Make Mistakes
How People Decide

Under each category are important points, described, buttressed by research, and boiled down into useful guidelines. This includes much of the research I talk about when I discuss deeper Instructional Design, and more. While it’s written for UI designers mostly, it’s extremely relevant to learning design as well. And it’s easy reading and reference, illustrated and to-the-point.

There are some really definitive books that people who design for people need to have read or have to hand. This fits into the latter category as does Dirksen’s book, while Don Norman’s books, e.g. Design of Everyday Things fit into the former. Must knows and must haves.

Meta-ethics of learning design

28 September 2021 by Clark 1 Comment

I’ve addressed ethics elsewhere, but I’m looking at it a different way now. I’m thinking from the perspective of situated cognition, and recognizing that there are certain things we can do. For better or worse. Further, these choices have ramifications beyond the initial impact. I think we need to be aware of the possibilities, and then consider the meta-ethics of learning design.

My particular concern sparks from the notion of how we are contextually sensitive. To make this concrete, think of the research by Beth Loftus. I”ll characterize a whole suite of research with a simply paraphrased experiment. So, she had folks watch a video of an accident. Then, she prompted recall of the amount of damage they witnessed. For one, she just asked them. For another group, the recall was prompted by “Besides the broken glass,…”. The latter group recalled worse damage. And. There. Was. No. Broken. Glass!

The point here is that context can influence our thinking and memory. Which is what I see worry about with videos. They can take a statement as if it’s fact, and then continue on with that as received wisdom. It’s a classic cognitive approach, making a statement as if it’s assumed.The bad part is that there’s a narrative flow, and it’s hard to stop and reflect. Versus, say, reading.

On the other hand, of course, we can build in reflection time. Sure, learners can use the pause button, but there are times they may not. For instance, if the learner is following confirmation bias, that is looking at things that align with what you believe.

We have the option to use coercive techniques for good, of course. However, we can also choose to use legitimate presentation techniques. I believe we should. Part of the development of metacognition is to see the pedagogy being used and internalizing it. If our pedagogy is visible (as it should be), it needs to be scrutable if we want our learners to adopt appropriate skills.

Our designs, and our meta-designs, need to be ethically designed, both to effectively achieve our ends, and to develop our learners. We need to support the meta-ethics of learning design, as well as the ethics themselves.

Complexity in Learning Design

21 September 2021 by Clark Leave a Comment

$a fractal$ I recently mentioned that one of the problems with research is that things are more interconnected than we think. This is particularly true with cognitive research. While we can make distinctions that simplify things in useful ways (e.g. the human information processing system model*), the underlying picture is of a more interactive system. Which underpins why it makes sense to talk about Learning Experience Design (LXD) and not just instructional design. We need to accommodate complexity in learning design. (* Which I talk about in Chapter 2 of my learning science book, and in my workshops on the same topic through the Allen Academy.)

We’re recognizing that the our cognition is more than just in our head. Marcia Conner, in her book Learn More Now mentioned how neuropeptides passed information around the body. Similarly, Annie Murphy Paul’s The Extended Mind talks about moving cognition (and learning) into the world. In my Make It Meaningful workshops (online or F2F at DevLearn 19 Oct), I focus on how to address the emotional component of learning. In short, learning is about more than just information dump and knowledge test.

Scientifically, we’re finding there are lots of complex interactions between the current context, our prior experience, and our cognitive architecture. We’re much more ‘situated’ in the moment than the rational beings we want to believe. Behavioral economics and Daniel Kahneman’s research have made this abundantly clear. We try to avoid the hard mental work using shortcuts that work sometimes, but not others. (Understanding when is an important component of this).

We get good traction from learning science and instructional design approaches, for sure. There are good prescriptions (that we often ignore, for reasons above) about what to do and how. So, we should follow them. However, we need more. Which is why I tout LXD Strategy! We need to account for complexity in learning design approaches.

For one, our design processes need to be iterative. We’ll make our best first guess, but it won’t be right, and we’ll need to tune. The incorporation of agile approaches, whether SAM or LLAMA or even just iterative ADDIE, reflects this. We need to evaluate and refine our designs to match the fact that our audience is more complex than we thought.

Our design also needs to think about the emotional experience as well as the cognitive experience. We want our design processes to systematically incorporate humor, safety, motivation, and more. Have we tuned the challenge enough, and how will we know? Have we appropriately incorporated story? Are our graphics aligned or adding to cognitive load? There are lots of elements that factor in.

Our design process has to accommodate SMEs who literally can’t access what they do. Also learner interests, not just knowledge. We need to know what interim deliverables, processes for evaluation, times when we shouldn’t be working solo, and tools we need. Most importantly, we have to do this in a practical way, under real-world resource constraints.

Which is why we need to address this strategically. Too many design processes are carry-over from industrial approaches: one person, one tool, and a waterfall process. We need to do better. There’s complexity in learning design, both on the part of our learners, and ourselves as designers. Leveraging what we know about cognitive science can provide us with structures and approaches that accommodate these factors. That’s only true, however, if we are aware and actively address it. I’m happy to help, but can only do so if you reach out. (You know how to find me. ;) Here’s to effective and engaging learning!

What about books | conferences?

18 May 2021 by Clark Leave a Comment

Responding to a frequent question yet again, I decided to post an answer to the “what about books | conferences?” question.

And, as usual, the transcript.

Once again, after talking about how learning requires meaningful practice, I was asked the seemingly timeless question: “but what about books?” Similarly, I regularly get “what about conferences?” So, for the record, let me say when and why books and lectures make sense. And when not. Hopefully I won’t have to answer another “what about books | conferences?” question.

To start, learning is action and reflection. That is, learning ‘outside’ formal instruction. We act in the world and reflect on it to cement the lesson. It’s slightly more complicated, because certain things, e.g. Geary’s biologically primary things, may not really need reflection. Further some things may be really challenging to learn on your own even with reflection. But basically, doing things and reflecting (which can be reading, experimenting, writing/representing), etc is the way we learn on our own.

Which, as I’ve argued before, suggests that instruction be designed action and guided reflection. That is, instructors should be choosing meaningful activities and scaffolding reflection around it. When we’re designing for novices [link], in particular, when the learner doesn’t know what’s important nor why, we need to do the whole enchilada (darn, now I’m hungry).

Which also means that when we’ve segued beyond novice to practitioner (and beyond), we begin to know what’s important and why, and we just need it. We want resources that can fill in the gaps. We want support for reflection.

So now we can explain why we can attend conferences, read books and articles, and the like. When we’re deeply engaged in something, whether work or a passion, reading a book, listening to someone tell their story, and the like, serves as the necessary adjunct to our activity! They provide the complement to our own endeavors; the reflection to our action!

Now, hopefully, we’ll never again need to discuss this. Realistically, we can point people here when we’ get “what about books | conferences”? At least, that’s my story, what’s yours?

A message to CxOs: about L&D myths

4 May 2021 by Clark 3 Comments

If you’re a CEO, COO, CFO, and the like, are you holding L&D to account? Because much of what I see coming out of L&D doesn’t stand up to scrutiny. As I’ve cited in books and presentations, there’s evidence that L&D isn’t up to scratch. And I think you should know a few things that may be of interest to you. So here’re some L&D myths you might want to watch out for.

If it looks like school, it must be learning. We’ve all been to school, so we know what learning looks like, right? Except, do you remember how effective school actually was? Did it give you many of the skills you apply in your job now? Maybe reading and writing, but beyond that, what did you learn about business, leadership, etc? And how did you learn those things? I’ll bet not by sitting and listening to lectures presented via bulletpoints. If it looks like schooling, it’s probably a waste of time and money. It should look more like lab, or studio.
If we’re keeping our efficiency in line with others, we’re doing good. This is a common belief amongst L&D: well, our [fill in the blank: employees served per L&D staff member | costs per hour of training | courses run per year | etc.] is the same or better than the industry average, so we’re doing good. No, this is all about efficiency, not effectiveness. If they’re not reporting on measurable changes in the improvement of business metrics, like sales, customer service, operations,e tc, they’re not demonstrating their worth. It’s a waste of money.
We produce the courses our customers need. Can they justify that? It’s a frequent symptom that the courses that are asked for have little relation to the actual problem. There are many reasons for performance problems, and a reliable solution is to throw a course at it. Without knowing whether it’s truly a function of lack of skill. Courses can’t address problems like the wrong incentives, or a lack of resources. If you’re not ensuring that you’re only using courses when they make sense, you’re throwing away money.
Job aids aren’t our job. Performance should be the job, not just courses. As Joe Harless famously said: “Inside every fat course there’s a thin job aid crying to get out.” There are many times when a job aid is a better solution than a course. To believe otherwise is one of the classic L&D myths. If they’re avoiding taking that on, they’re avoiding a cheaper and more effective solution.
Informal learning isn’t our job. Well, it might not be if L&D truly doesn’t understand learning, but they should. When you’re doing trouble-shooting, research, design, etc., you don’t know the answer when you start. That’s learning too, and there is a role for active facilitation of best principles. Assuming people know how to do it isn’t justifiable. Informal learning is the key to innovation, and innovation is a necessary differentiation.
Our LMS is all we need. Learning management systems (which is a misnomer, they’re course management systems) manage courses well. However, if they’re trying to also be resource portals, and social media systems, and collaboration tools, they’re unlikely to be good at all that. Yet those are also functions that affect optimal performance and continual innovation (the two things I argue should be the remit of L&D). Further, you want the right tool for the job. One all-singing, all-dancing solution isn’t the way to bet for IT in general, and that holds true for L&D as well.
Our investment in evaluation instruments is valuable. If you’re using some proprietary tools that purport to help you identify and characterize individuals, you’re probably being had. If you’re using it for hiring and promotion, you’re also probably violating ethical guidelines. Whether personality, or behavior, or any other criteria, most of these are methodologically and psychometrically flawed. You’re throwing away money. We have a natural instinct to categorize, but do it on individual performance, not on some flawed instrument.
We have to jump on this latest concept. There’re a slew of myths and misconceptions running around that are appealing and yet flawed. Generations, learning styles, attention spans, neuro-<whatever> and more are all appealing, and also misguided. Don’t spend resources on investing in them without knowing the real tradeoffs and outcomes.These are classic L&D myths.
We have to have this latest technology. Hopefully you’re resistant to new technologies unless you know what they truly will do for your organization. This holds true for L&D as well. They’re as prone to lust after VR and AR and AI as the rest of the organization. They’re also as likely to spend the money without knowing the real costs and consequences. Make sure they’re coming from a place where they know the unique value the technology brings!

There’s more, but that’s enough for now. Please, dig in. Ask the hard questions. Get L&D to be scrutable for real results, not platitudes. Ensure that you’re not succumbing to L&D myths. Your organization needs it, and it’s time to hold them to account as you do the rest of your organization. Thanks, and wishing you all the best.

Something that emerged from a walk, and, well, I had to get it off my chest. I welcome your thoughts.

More Marketing Myths

24 June 2020 by Clark 1 Comment

The other day a tweet caught my eye that used a myth to get you to click a link. Worse, clicking the link led to another myth. These are folks I think are generally good, and it seems that their actual offering made sense, but the approach does not. It’s just more marketing myths, drawing upon common misconstruals, and that’s not a good thing. I think it’s worth calling out.

It’s like the claim that we’ve dropped to the ‘attention span of a goldfish’ to argue for shorter ads, learning, etc. That, of course, was a misinterpretation of data, with only a couple of implausibilities. First, how do you measure the attention span of a goldfish? More importantly, how would we evolutionarily change our cognitive architecture in the span of a few years? Er, no. Attention’s complex, and the argument is spurious. (Ever disappear into a novel/game/book and surface several hours later wondering where the time went? Yeah, that.)

In this case, they were supporting a proposition that makes sense. Instead of just having a research report, they’re suggesting a summary video. Yep, having a dynamic visual presentation of data is a supportable argument. As Jill Larkin & Herb Simon argued in their Cognitive Science article, Why A Diagram is (Sometimes) Worth Ten Thousand Words, it’s about mapping conceptual relationships to spatial relationships to allow our strong spatial processing to assist.

What was wrong was that the lead-in in the tweet was saying “65% of the population are visual learners.” Er, no. That’s the learning styles myth. First, there aren’t reliable instruments. Further, people change depending on what’s to be learned, why, where, when, and more. We learn visually, and auditorily, and kinesthetically, and… Suggesting that we know how many people are a type of learner is basically wrong.

They went on to say “..enhances knowledge delivery by catering to how people want to learn.” There’s research to say that’s not a good basis, either. The relationship between people’s preferences and what’s effective is pretty close to zero.

It gets worse. When I clicked on the link, it took me to this claim: “Studies show that people remember…10% of what they hear, 20% of what they read, 80% of what they see and do”. Nope. That’s the Dale’s Cone myth, and that’s been shown to be made up. You should indeed remember more of what you do, but reading is seeing, and depending on context, hearing might be best, and…This is even worse, conflating seeing and doing. Dale never added numbers, and the numbers just aren’t plausible anyways, being too perfect.

So, really, three myths for the price of two. It’s more marketing myths in service of selling you things. The most important part is that you don’t need to do this! There are perfectly good, comprehensible ways to push this message without relying on myths. Please, be wary, be leery, be a skeptical consumer. Caveat emptor!