Learnlets

Secondary

Clark Quinn’s Learnings about Learning

Quality or Quantity?

by 4 Comments

Recently, there’s been a lot of excitement about Generative Artificial Intelligence (Generative AI). Which is somewhat justified, in that this technology brings in two major new capabilities. Generative AI is built upon a large knowledge base, and then the ability to generate plausible versions of output. Output can in whatever media: text, visuals, or audio. However, there are two directions we can go. We can use this tool to produce more of the same more efficiently, or do what we’re doing more effectively. The question is what do we want as outcomes: quality or quantity?

There are a lot of pressures to be more efficient. When our competitors are producing X at cost Y, there’s pressure to do it for less cost, or produce more X’s per unit time. Doing more with less drives productivity increases, which shareholders generally think are good. There’re are always pushes for doing things with less cost or time. Which makes sense, under one constraint: that what we’re doing is good enough.

If we’re doing bad things faster, or cheaper, is that good? Should we be increasing our ability to produce planet-threatening outputs? Should we be decreasing the costs on things that are actually bad for us? In general, we tend to write policies to support things that we believe in, and reduce the likelihood of undesirable things occurring (see: tax policy). Thus, it would seem that if things are good, go for efficiency. If things aren’t good, go for quality, right?

So, what’s the state of L&D? I don’t know about you, but after literally decades talking about good design, I still see way too many bad practices: knowledge dump masquerading as learning, tarted up drill-and-kill instead of skill practice, high production values instead of meaningful design, etc. I argue that window-dressing on bad design is still bad design. You can use the latest shiny technology, compelling graphics, stunning video, and all, but still be wasting money because there’s no learning design underneath it.  To put it another way, get the learning design right first, then worry about how technology can advance what you’re doing.

Which isn’t what I’m seeing with Generative AI (as only the latest in the ‘shiny object’ syndrome. We’ve seen it before with AR/VR, mobile, virtual worlds, etc. I am hearing people saying “how can I use this to work faster”,  put out more content per unit time”, etc, instead of “how can we use this to make our learning more impactful”. Right now, we’re not designing to ensure meaningful changes, nor measuring enough of whether our interventions are having an impact. I’ll suggest, our practices aren’t yet worth accelerating, they still need improving! More bad learning faster isn’t my idea of where we should be.

The flaws in the technology provide plenty of fodder for worrying. They don’t know the truth, and will confidently spout nonsense. Generative AIs don’t ‘understand’ anything, let alone learning design. They are also knowledge engines, and can’t create impactful practice that truly embeds the core decisions in compelling and relevant settings. They can aid this, but only with knowledgeable use. There are ways to use such technology, but it comes from starting with the point of actually achieving an outcome besides having met schedule and budget.

I think we need to push much harder for effectiveness in our industry before we push for efficiency.  We can do both, but it takes a deeper understanding of what matters. My answer to the question of quality or quantity is that we have to do quality first, before we address quantity. When we do, we can improve our organizations and their bottom lines. Otherwise, we can be having a negative impact on both. Where do you sit?

One may not be enough

by Leave a Comment

A recent intersection of talks leads to an interesting issue for L&D. First, we recently talked to Guy Wallace about his recent book, The L&D Pivot Point. Then, we talked to Julie Dirksen about her new book, Talk to the Elephant. The interesting thing is that there’s some overlap between the two ideas that isn’t immediately obvious, but really important. The realization is that when we’re talking about barriers to success, thinking of one may not be enough.

So, Guy’s book is about taking a step above just thinking of course. He’s a proponent of performance improvement consulting, where you analyze the problem before you decree a course as a solution. The important recognition is that there can be multiple barriers to performance, including a lack of skills indicating a course. However, other reasons might be the wrong incentives, a lack of resources, etc. Sometimes a job aid can do better, some times neither that or a course will suffice.

Julie’s book, on the other hand, is a complement to her first book, Design for How People Learn. She recognized that even good design (what her first book did, eloquently) might not help learning stick, and looked at other barriers, such as managers extinguishing the learning. She was more focused on making the learning design succeed.

What she did, however, is provide a rich suite of potential barriers, along with solutions, and suggest that you may need to address more than one. That goes along with, and complements, Guy’s focus.

Just as you design programs that include messaging, training, support, rewards, and more, you should also ensure that you’ve analyzed all the barriers to performance. You might address learning, provide job aids, ensure incentives are aligned, prepare supervisors, and more. Addressing only a particular situation may not be sufficient. You may have several barriers, When it comes to solutions, one may not be enough. This argues (again) for rigorous analysis and a success focus, not just doing what you are comfortable with. In the long term, I reckon this is where we need to go as we move from learning to performance (and development). your thoughts?

Where are we at?

by 1 Comment

Signs pointing multiple directions with distances. I was talking with a colleague, and he was opining about where he sees our industry. On the other hand,  had some different, and some similar thoughts. I know there are regular reports on L&D trends, with greater or lesser accuracy. However, he was, and I similarly am looking slightly larger than just “ok, we’re now enthused about generative AI“. Yes, and, what’s that a signal of? What’s the context? Where are we at?

When I’m optimistic, I think I see signs of an awakening awareness. There are more books on learning science, for instance. (That may be more publishers and people looking for exposure, but I remain hopeful.)  I see a higher level of interest in ‘evidence-based’. This is all to the good (if true). That is, we could and should be beginning to look at how and why to use technology to facilitate learning appropriately.

On the cynical side, of course, is other evidence. For example, the interest in generative AI seems to be about ways to reduce costs. That’s not really what we should be looking at. We should be freeing up time to focus on the more important things, instead of just being able to produce more ‘content’ with even less investment. The ‘cargo cult’ enthusiasm about: VR, AR, AI, etc still seems to be about chasing the latest shiny object.

As an aside, I’ll still argue that investing in understanding learning and better design will have a better payoff than any tech without that foundation. No matter what the vendors will tell you!  You can have an impact, though of course you risk having a previous lack of impact exposed…

So, his point was that he thought that more and more leaders of L&D are realizing they need that foundation. I’d welcome this (see optimism, above ;).  Similarly, when I argue that if Pine & Gilmore are right (in The Experience Economy) as to what’s the next step, we should be the ones to drive the Transformation Economy (experiences that transform you).  Still,  is this a reliable move in the field? I still see folks who come in from other areas of the biz to lead learning, but don’t understand it. I’ll also cite the phenomena that when folks come into a new role they need to be seen to be doing something. While them getting their mind around learning would be a good step, I fear that too many see it as just management & leadership, not domain knowledge. Which, reliably, doesn’t work. Ahem.

Explaining the present, let alone predicting the future, is challenging. (“Never predict anything, particularly the future!”) Yet, it would help to sort out whether there is (finally) the necessary awakening. In general, I’ll remain optimistic, and continue to push for learning science, evidence, and more. That’s my take. What’s yours? Where are we at?

The Pivotal Point

by 3 Comments

We (the Learning Development Accelerator) just released Guy Wallace’s latest tome, The L&D Pivot Point. Then, we had an interview with him to explain what it’s about. Despite having a ring-side seat (I served as editor, caveat emptor), it was eye-opening to hear him talk about what it’s about! It really is about the pivotal point in L&D, when you move from just offering courses to looking at performance. It’s such an important point that it’s worth reiterating.

So, the official blurb for the book talks about his tried and tested processes. In the interview, he talks about how he’s synthesized the work of the leaders of the performance improvement movement, people like Joe Harless, Geary Rummler, Thomas Gilbert, Robert Mager, Thiagi, and more. While the models they used differed, Guy’s created a synthesis that makes sense, and more importantly, works. He talked about how he refined his work to balance effectiveness with efficiency. Moreover, his approach avoids any redundant work.

Interestingly, he also recounted how his approach achieved buy-in from the stakeholders to the extent that he had to fight to not keep them all on the team through all the stages! That’s a great outcome, and it comes from demonstrating value. He focuses on where performance needs are critical, and thus it has a natural interest, but too many of the approaches can stifle that interest. Instead, his intent focus on meaningful outcomes truly engages everyone from the performers to the executives.

Guy also is quite open about the problems facing our industry. Despite the necessity of starting as order takers (essentially, “you can’t say ‘no'”), he estimates that only 20% of the time is the problem a learning or skills problem. Which resonates with other data I’ve seen about the value of training interventions! Instead, there can be many drivers for problems in performance.  His approach includes detailed analyses that identify the root cause of the problem, and when to determine that it’s worth trying an intervention. He’s quite open about how that can lead to a shift in intervention focus. At other times, it might lead to a hiatus while problems get attention.

One other thing I found interesting in the interview was how he talked about potential barriers to success up front. While it might seem like a deterrent, he pointed out how it led to making sense later. That is, folks would soon see that, for instance, supervisor support was critical to success. He includes a rigorous analysis of potential barriers as part of the book.

Quite simply, L&D has a problem of going from go-to-whoa without considering whether a course is the right solution. Guy’s book is a way to avoid doing that, and systematically evaluating what the pivotal point should be for determining whether we can successfully intervene or not, and how. There’s much more: how to manage the process, deal with stakeholders, and test your assumptions. It’s in his own inimitable style (lessons learned on editing ;), but there’s deep wisdom there. That’s my take, at least, I welcome yours.

Getting Engagement Right

by Leave a Comment

I’m on record stating that I think learning experience design (LXD) is the elegant integration of learning science and engagement. In addition, I’ve looked at both. Amongst the things that stand out for me are that there are an increasing suite of resources for learning science. For one, I have my own book on the topic! There are other good ones too. However, on the flip side, for engagement, I didn’t find much. I had intended to write an LXD book, but then ATD asked for the learning science one. Once it was done, however, I quickly realized that I wanted to write the complement. Thus, Make It Meaningful was born. It’s available, but I’m also running a workshop on the topic, starting this coming week! Four weekly 2 hour meetings, at the convenient time of noon ET. It’s all about getting engagement right. So, what’s covered?

For the first week, there’s an overview of the importance of engagement, and how to set the ‘hook’. We’ll briefly review the reasons why to consider the engagement side (and trust me, this is something you want to do). Then we’ll talk about the first step, getting learners to the a motivated state to begin the learning. We’ll look at barriers to success as well, and what to do.

In the second week, we’ll talk about ‘landing’ the experience. Once the hook is set, it doesn’t mean you’ve got them through the experience. Instead, there’s much to do to maintain that motivation. In addition, you want to ensure that anxiety doesn’t overwhelm the learning, and you want to build confidence. We’ll talk about principles as well as heuristics.

In the third week, we dig into what this means in practical terms. What is an engaging introduction? What about the models and examples? The critical element is the practice that learners perform. We’ll talk about how aligning the practice with the desired objectives while making a compelling context is necessary, but doable.

In the last week, we’ll talk about making a design process that can reliably deliver on learning experience. We’ll take a generic design process and go through the changes that ensure both an effective learning design and an engaging experience. We’ll work from analysis, through specification, and on to evaluation (we won’t talk much about implementation, because of my quip that getting the design right leaves lots of ways to create the solution, and not doing so renders everything else extraneous.

Sure, you can just buy the book, and that’s ok. I’m all about getting the word out, and getting better learning happening for our learners. However, in the workshop, not only do you get the book, but we’ll work through the ideas systematically, put them into practice, and address the individual questions you may have. Look, getting engagement right is an advanced topic, but it’s increasingly what will differentiate our solutions from the knowledge ones that come from typical approaches, no matter how technologically augmented. This stuff matters! So, I hope to see you there.

DnD n LnD

by 2 Comments

multi-sided diceLast Friday, I joined in on a Dungeons & Dragons (DnD) campaign. This wasn’t just gratuitous fun, however, but was explicitly run to connect to Learning & Development folks (LnD). Organized by the Training, Learning, and Development Community (a competitor to LDA? I have bias. ;), there was both some preliminary guidance, and outcomes. I was privileged to play a role, and while not an official part of the followup (happening this week), I thought I’d share my reflections.

So, first, my DnD history. I played a few times while in college, but… I gave it up when a favorite character of mine was killed by an evil trap (that was really too advanced for our party). I’ve played a lot of RPGs since then, with a lot of similarities to the formal DnD games (tho’ the actual ones are too complex). Recently, with guidance from offspring two, our family is getting back into it (with a prompt from a Shakespeare and DnD skit at the local Renaissance Faire).

Then, I’ve been into games for learning since my first job out of college, programming educational computer games. It also became the catalyst for my ongoing exploration of engagement to accompany my interest in cognition/learning, design, and technology. The intersection of which is where I’ve pretty much stayed (in a variety of roles), since then! (And, led to my first book on how to do same.)

Also, about DnD. It’s a game where you create a character. There are lots of details. For one, your characteristics: strength, dexterity, wisdom, intelligence, and more. Those combine with lots of attributes (such race & role). Then, there’s lots of elaboration: backstory, equipment, and more. This can alter during the game, where your abilities also rise. This adds complexity to support ongoing engagement. (I heard one team has been going for over 40 years!)

Characters created by the players are then set loose in a campaign (a setting, precipitating story, and potential details). A Dungeon Master runs the game, Keegan Long-Wheeler in our case, writing it and managing the details. Outcomes happen probabilistically by rolling dice. Computers can play a role. For one, through apps that handle details like rolling the dice. Then folks create games that reflect pre-written campaigns.

One important thing, to me, is that the players organize and make decisions together. We were a group who didn’t necessarily know each other, and we were playing under time constraints. This meant we didn’t have the dialog and choices that might typically emerge in such playing. Yet, we managed a successful engagement in the hour+ we were playing. And had fun!

I was an early advocate of games for learning. To be clear, not the tarted up drill and kill we were mostly doing, but inspired by adventure games. John Carroll had written about this back in the day, I found out. However, I’d already seen adventure games having the potential to be a basis for learning. Adventure games naturally require exploring. In them, you’re putting clues together to choose actions to overcome obstacles. Which, really, is good learning practice! That is, making decisions in context in games is good practice for making decisions in performance situations. Okay, with the caveat that you should design the game so that decisions have a natural embed.

The complexity of DnD is a bit much, in my mind, for LnD, but…the design!  The underlying principles of designing campaigns bears some relation to designing learning experiences. I believe designing engaging learning may be harder than designing learning or games, but we do have good principles. I do believe learning can, and should, be ‘hard fun‘.  Heck, it’s the topic of my most recent tome! (I believe learning should be the elegant integration of learning science with engagement.)

This has been an opportunity to reflect a bit on the underlying structure of games, and what makes them work. That’s always a happy time for me. So, I’m curious what you see about the links between games and learning!

Bad research

by 1 Comment

How do you know what’s dubious research? There are lots of signals, more than I can cover in one post. However, a recent discovery serves as an example to illustrate some useful signals. I was trying to recall a paper I recently read, which suggested that reading is better than video for comprehending issues. Whether that’s true or not isn’t the issue. What is the issue is that in my search, I came across an article that really violated a number of principles. As I am wont to do, let’s briefly talk about bad research.

The title of the article (paraphrasing) was “Research confirms that video is superior to text”. Sure, that could be the case! (Actually the results say, not surprisingly, that one media’s better for some things, and another’s better at other; BTW, one of our great translators of research to practice, Patti Shank, has a series of articles on video that’s worth paying attention to.) Still, this article claimed to have a definitive statement about at least one study. However, when I looked at it, there were several problems.

First, the study was a survey asking instructors what they thought of video. That’s not the same as an experimental study! A good study would choose some appropriate content, and then have equivalent versions in text and video, and then have a comprehension test. (BTW, these experiments have been done.) Asking opinions, even of experts, isn’t quite as good. And these weren’t experts, they were just a collection of instructors. They might have valid opinions, but their expertise wasn’t a basis for deciding.

Worse, the folks conducting the study had. a. video. platform.  Sorry, that’s not an unbiased observer. They have a vested interest in the outcome. What we want is an impartial evaluation. This simply couldn’t be it. Not least, the author was the CEO of the platform.

It got worse. There was also a citation of the unjustified claim that images are processed 60K times better than text, yet the source of that claim hasn’t been found! They also cited learning styles! Citing unjustified claims isn’t a good practice in sound research. (For instance, when reviewing articles, I used to recommend rejecting them if they talked learning styles.) Yes, it wasn’t a research article on it’s own, but…I think misleading folks isn’t justified in any article (unless it’s illustrative and you then correct the situation).

Look, you can find valuable insights in lots of unexpected places, and in lots of unexpected ways. (I talk about ‘business significance’ can be as useful as statistical significance.) However, an author with a vested interest, using an inappropriate method, to make claims that are supported by debunked data, isn’t it. Please, be careful out there!

Get the basics right first!

by Leave a Comment

I’m currently advising several organizations on their approaches to the use of technology to support learning. Moreover, I’ve been doing so for more than two decades, and see a lot more such situations as well. One of the things that I struggle with is seeing folks getting all agog over new technology, yet without getting the design right beforehand.  Thus, let me make a simple suggestion: get the basics right first!

So, we know what leads to good learning. Heck, I’ve written a book summarizing what’s known about it, and I’m not the only one. Despite that fact that humans are complex, and increasingly so are our learning goals, there exist robust principles. We know that we should provide a sufficient quantity of appropriately challenging contextualized practice with aligned feedback, for instance. That is, if we actually want to achieve an outcome.

Yet, too often, we don’t see this. We see, instead, information presentation. Sometimes even with a knowledge test! Yet, such an effort is unlikely to lead to any meaningful change. That is, the investment’s wasted!

Worse, too often we see this being done with fancy new tools. Sure, I get as attracted to shiny new objects as anyone. However, I want to understand their core affordances for learning. Anyone had the dubious pleasure of attending a slide presentation in a virtual world? Or maybe being presented with animated presentations of lots of facts? The new tools may have a short-term effect of novelty, but that’s it. The fundamental aspects of how our brains learn are what’s going to make, or break, a learning investment.

On the other hand, if we start with getting the learning right, first, then there may be additional value coming from the tech. Adaptivity, on top of quality learning design, can accelerate the outcomes.  Immersion, at the right time and place, is better than not. Language models, properly used, can have big impacts. However, it comes from knowing the specific capabilities, and matching them to the need.

While I haven’t done the ‘back of the envelope’ calculation (I’m not a financial whiz), I can state with a fair degree of comfort that you’re better off doing simple learning with good design. Bad design with shiny tech is still bad design! You’ll more likely have an impact putting your investment into learning quality than using fancy tech to deliver dreck. Of course, once you’ve done that, the investment in tech can do a lot more!

I’m not against new tech, heck I’ve written on games, mobile, and more! What I’m against is new tech in lieu of good design. And I’m even more enamored of good tech on top of good design.  So, get the basics right first, then add in the shiny objects. That way you’re going to have a good return on your $$, and that’s a good thing. Right?

PS, speaking of basics, we’ll be running a debate tomorrow (11 Oct) discussing the Learning Experience Design (LXD) label. I’m sure we’ll unpack critical issues. Check it out. 

Talking ‘transfer’

by Leave a Comment

I was looking to wrap up my slide deck for DevLearn (will be talking learning science), and wanted to see if there was anything missing. In particular, I knew I had a fair bit on retention, but I was checking on what I call ‘transfer’. In doing so, I realized that colleagues and others use the term differently! Which means it’s time to talk ‘transfer’.

So one of our good research translators is Will Thalheimer. I found his report (PDF) on transfer, and its usage was as many others I’d found. Will describes transfer as moving learning from the instructional environment to the workplace. No argument this is important! Yet, it’s not how I use the word.

In talking learning science, I’ve seen and said that the two gold standards are retention over time until needed, and transfer to all appropriate (and no inappropriate) situations. (That’s what I’ll be talking about in my session: what leads to those.) My notions here are both part of his transfer.

I get my usage of transfer essentially from folks like John Anderson at Carnegie Mellon Uni, who talks about what facilitates transfer. He’s the one I think of, for instance, when thinking of identical elements transfer, which stipulates that the more elements are the same, the higher the transfer. There’s near and far transfer, as well. That indicates the breadth of area you’re transferring to, really. E.g. negotiation would be far transfer, as there are many situations, whereas training to specifically run projector MX3600D4 is very much near transfer.

In a review of transfer I found (PDF), a distinction is made between ‘low-‘ and ‘high-road’ transfer. The former ‘happens’, while the latter is engineered. Given that, to me, instruction is about engineering success, maximizing likelihoods, I’m all for the latter. So, choosing contexts and supporting reflection.

To memory, even the academic literature about workplace learning seems to support the other interpretation. Yet, my term seems more in line with what the learning science community. In the longer term, we’ll need to reconcile, but for me, the distinction between the elements of retention and transfer are important. So, when I’m talking ‘transfer’, what I mean is the breadth of application. That’s my take, what are your thoughts?

Not diplomatic, but do care!

by Leave a Comment

So, I’ve been let know that my feedback is ‘blunt’. Which, I suppose, is better than some other alternatives. Still, I’d at least like to contextualize it, since I really can’t deny it. Hey, other people’s opinions are valid! They may not be correct, but I’m willing to acknowledge that it’s probably true. So, I guess I want to say I am not diplomatic, but do care.

So, I can be a little obtuse. I don’t always pick up on the subtleties of human interaction. Not a lot of evidence it’s actually worth a label, but… I recall going to meetings with my clients, who are executives at the C-level. We’d come out and they’d ask me “what did you see?” I’d respond with an analysis of the performance problem, and possible paths. I’d return the question, and hear things like “that’s the person with the real power”, “that one doesn’t like this other”, and other such perceptions. I just don’t see that stuff!

Even back when I was supervising students, I remember a couple of them saying “we never have any doubt about whether you’re not happy with something”. #oops On the other hand, I also typically don’t hear people saying I’m deceptive, or misleading. My opinions are pretty clear, but so too, generally, are my motives. (Because I’m inept at hiding them, I would suspect!) I do strive to make my feedback instructive as well as constructive. I perhaps over-err on the side of explaining why I’m saying something, and try to remember not to offer advice to fix it unless asked. (Not perfectly, by any means!)

I do also recognize that I may not always contextualize my feedback. In particular, I can forget to say what’s good before I say what should be fixed. I could defend myself and say that I don’t critique unless it’s worth fixing, but really I shouldn’t assume they know that. I am not deliberately mean, and never mean to hurt anyone’s feelings; I try to recognize that most folks don’t have bad intentions. And, of course, I don’t always succeed. Similarly, I try to admit my flaws, but again, don’t always succeed.

So, for the record, I care very much. And, we can be awesome! So, I do my best to share what I’ve learned. I freely admit my total luck at having opportunities to be the right place and have exposure to some great minds. I’ve had to work hard to learn it, mostly because it came easy early on. Then at the higher levels I learned that I couldn’t just wing it anymore.

So, my main point here is that when you see me critique things, it’s because it can be better. I do it publicly (though usually anonymously), so we can all learn. And I welcome feedback and pushback! I have lots to learn, and always will.

Oh, and please, please don’t be intimidated to talk to me. I’m really not an ogre. (I am a bit of an introvert, so I can seem standoffish, but it’s really just insecurity ;). Admittedly, I can have a very short fuse for folks trying to take advantage of me (or others), but… I really do care, and want to help everyone. I just have a responsibility to allocate my time in the most efficient ways. As I’ve said before, I talk ideas for free; I help someone personally for drinks/dinner; if someone’s making a quid, I deserve a cut. Until I hear otherwise, that’s seemed fair for close on two decades. So me: not diplomatic, but do care. Fair enough?