Learnlets

Secondary

Clark Quinn’s Learnings about Learning

The case for model answers (and a rubric)

by 4 Comments

Human body modelAs I‘ve been developing online workshops, I‘ve been thinking more about the type of assessment I want. Previously, I made the case for gated submissions. Now I find another type of interaction I‘d like to have. So here‘s the case for model answers (and a rubric).

As context, many moons ago we developed a course on speaking to the media. This was based upon the excellent work of the principals of Media Skills, and was a case study in my  Engaging Learning book. They had been running a face to face course, and rather than write a book, they wondered if something else could be done. I was part of a new media consortium, and was partnered with an experienced CD ROM developer to create an asynchronous elearning course.  

Their workshop culminated in a live interview with a journalist. We couldn‘t do that, but we wanted to prepare people to succeed at that as an optional extra next step. Given that this is something people really fear (apocryphally more than death), we needed a good approximation. Along with a steady series of exercises going from recognizing a good media quote, and compiling one, we wanted learners to have to respond live. How could we do this?

Fortunately, our tech guy came up with the idea of a programmable answering machine. Through a series of menus, you would drill down to someone asking you a question, and then record an answer. We had two levels: one where you knew the questions in advance, and the final test was one where you‘d have a story and details, but you had to respond to unanticipated questions.  

This was good practice, but how to provide feedback? Ultimately, we allowed learners to record their answers, then listen to their answers and a model answer. What I‘d add now would be a rubric to compare your answer to the model answer, to support self-evaluation. (And, of course, we’d now do it digitally in the environment, not needing the machine.)

So that‘s what I‘m looking for again. I don‘t need verbal answers, but I do want free-form responses, not multiple-choice. I want learners to be able to self-generate their own thoughts. That‘s hard to auto-evaluate. Yes, we could do whatever the modern equivalent to Latent Semantic Analysis is, and train up a system to analyze and respond to their remarks. However, a) I‘m doing this on my own, and b) we underestimate, and underuse, the power of learners to self-evaluate.  

Thus, I‘m positing a two stage experience. First, there‘s a question that learners respond to. Ideally, paragraph size, though their response is likely to be longer than the model one; I tend to write densely (because I am). Then, they see their answer, a model answer, and a self-evaluation rubric.  

I‘ll suggest that there‘s a particular benefit to learners‘ self-evaluating. In the process (particularly with specific support in terms of a mnemonic or graphic model), learners can internalize the framework to guide their performance. Further, they can internalize using the framework and monitoring their application to become self-improving learners.

This is on top of providing the ability to respond in richer ways that picking an option out of those provided. It requires a freeform response, closer to what likely will be required after the learning experience. That‘s similar to what I‘m looking for from the gated response, but the latter expects peers and/or instructors to weigh in with feedback, where as here the learner is responsible for evaluating. That‘s a more complex task, but also very worthwhile if carefully scaffolded.  

Of course, it‘d also be ideal if an instructor is monitoring the response to look for any patterns, but that‘s outside the learners‘ response. So that‘s the case for model answers. So, what say you? And is that supported anywhere or in any way you know?

How to be an elearning expert

by 3 Comments

I was asked (and have been a time or two before): “What’s the one most important thing you’d like to tell to be successful Ed Tech industry leader” Of course there wasn‘t just one ;). Still, looking at colleagues who I think fit that characterization, I find some commonalities that are worth sharing. So here‘s one take on how to be an elearning expert.

Let‘s start with that ‘one thing‘.   Which is challenging, since it‘s more than one thing! Still, I boiled it down into two components: know your stuff, and let people know.   That really is the core. So let‘s unpack that some more.   The first thing is to establish credibility. Which means demonstrating that you track and promote the right stuff.  

Some folks have created a model that they tout. Cathy Moore has Action Mapping, Harold Jarche has PKM, Con Gottfredson has the 5 moments of need, and so on.   It‘s good having a model, if it‘s a good, useful one (there are people who push models that are hype or ill-conceived at best). Note that it‘s not necessarily the case that these folks are just known for this model, and most of these folks can talk knowledgeably about much more, but ‘owning‘ a model that is useful is a great place to be. (I occasionally regret that I haven‘t done a good job of branding my models.) They understand their model and its contribution, it‘s a useful one, and therefore they contribute validly that way and are rightly recognized.

Another approach like this is owning a particular domain. Whether gaming (e.g. Karl Kapp), visuals (Connie Malamed), design (Michael Allen), mixed realities (Ann Rollins), AI (Donald Clark), informal (Jane Hart), evaluation (Will Thalheimer), management (Matt Richter), and so on, they have deep experience and a great conceptual grasp in a particular area. Again, they can and do speak outside this area, but when they talk about these topics in particular, what they say is worthy of your attention!

Then there are other folks who don‘t necessarily have a single model, but instead reliably represent good science. Julie Dirksen, Patti Shank, Jane Bozarth, Mirjam Neelen, and others  have established a reputation for knowing the learning science and interpreting it in accurate, comprehensible, and useful ways.  

The second point is that these folks write and talk about their models and/or approaches. They‘re out there, communicating. It‘s about reliably saying the important things again and again (always with a new twist). A reputation doesn‘t just emerge whole-cloth, it‘s built step by step. They also practice what they preach, and have done the work so they can talk about it. They talk the talk and walk the walk. Further, you can check what they say.  

So how to start? There are two clear implications. Obviously, you have to Know. Your. Stuff! Know learning, know design, know engagement, know tech. Further, know what it means in practice!   You can focus deeply in one area, or generate one useful and new model, or have a broad background, but it can‘t just be in one thing. It‘s not just all your health content for one provider. What you‘re presenting needs to be representative and transferable.  Further, you need to keep up to date, so that means continually learning: reading, watching, listening.

Second, it‘s about sharing. Writing and speaking are the two obvious ways. Sure, you can host a channel: podcast, vlog, blog, but if you‘re hosting other folks, you‘re seen as well connected but not necessarily as the expert. Further, I reckon you have to be able to write and speak (and pretty much all of these folks do both well).   So, start by speaking at small events, and get feedback to improve. Study good presentation style. Then start submitting for events like the Learning Guild, ATD, or LDA (caveats on all of these owing to various relationships, but I think they‘re all scrutable). I once wrote about how to read and write proposals, and I think my guidance is still valid.

Similarly, write. Learning Solutions or eLearn Mag are two places to put stuff that‘s sensibly rigorous but written for practitioners.   Take feedback to heart, and deliberately improve. Make sure you‘re presenting value, not pitching anything. What conferences and magazines say about not selling, that your clear approach is what sells, is absolutely true.  

Also, make sure that you have a unique ‘voice’. No one needs the same things others are saying, at least in the same way. Have a perspective, your own take. Your brand is not only what you say, but how you say it.

A related comment: track some related fields. Most of the folks I think of as experts have some other area they draw inspiration from. UX/UI, anthropology, software engineering, there are many fields and finding useful insight from a related one is useful to the field and keeps you fresh.

Oh, one other thing. You have to have integrity. People have to be able to trust what you say. If you push something for which you have a private benefit, or something that‘s trendy but not real, you will lose whatever careful credibility you‘ve built up. Don‘t squander it!  

So that‘s my take on how to be an elearning expert. So, what have I missed?

A message to CxOs 2: about org learning myths

by 2 Comments

When I wrote my last post on a message to CxOs about L&D myths, I got some pushback. Which, for the record, is a good thing; one of us will learn something. As a counter to my claim that L&D often was it’s own worst enemy, there was a counter. The claim was that there are folks in L&D who get it, but fight upward against wrong beliefs. Which absolutely is true as well. So, let‘s also talk about what CxOs need to know about the org learning myths they may believe.  

First, however, I do want to say that there is evidence that L&D isn‘t doing as well as it could and should. This comes from a variety of sources. However, the question is where does the blame lie. My previous post talked about how L&D deludes itself, but there are reasons to also believe in unfair expectations. So here‘s the other side.  

  1. If it looks like schooling… I used this same one against L&D, but it‘s also the case that CxOs may believe this. Further, they could be happy if that‘s the case. Which would be a shame just as I pointed out in the other case. Lectures, information dump & knowledge test, in general content presentation doesn‘t lead to meaningful change in behavior in the absence of activity. Designed action and guided reflection, which looks a lot more like a lab or studio than a classroom, is what we want.
  2. SMEs know what needs to be learned. Research tells us to the contrary; experts don’t have conscious access to around 70% of what they  do (tho’ they do have access to what they know). Just accepting what a SME says and making content around that is likely to lead to a content dump and lack of behavior change. Instead, trust (and ensure) that your designers know more about learning than the SME, and have practices to help ameliorate the problem.
  3. The only thing that matters is keeping costs low.  This might seem to be the case, but it reflects a view that org learning is a necessary evil, not an investment. If we’re facing increasing change, as the pundits would have it, we need to adapt. That means reskilling. And effective reskilling isn’t about the cheapest approach, but the most effective for the money. Lots of things done in the name of learning (see above) are a waste of time and money. Look for impact first.
  4. Courses are the answer to performance issues.  I was regaled with a tale about how sales folks and execs were  insisting that customers wanted training. Without evaluating that claim. I’ll state a different claim: customers want solutions. If it’s persistent skills, yes, training’s the answer. However, a client found that customers were much happier with how-to videos than training for most of the situations. It’s a much more complex story.
  5. Learning stops at the classroom. As is this story. One of the reasons Charles Jennings was touting 70:20:10 was not because of the numbers, but because it was a way to get execs to realize that only the bare beginning came from courses, if at all. There’s ongoing coaching with stretch assignments and feedback, and interacting with other practitioners…don’t assume a course solves a problem. A colleague mentioned how her org realized that it couldn’t create a course without also creating manager training, otherwise they’d undermine the outcomes instead of reinforcing them.
  6. We‘ve invested in an LMS, that‘s all we need. That’s what the LMS vendors want you to believe ;)!  Seriously, if all you’re doing is courses, this could be true, but I’m hoping the above
  7. Customers want training.  Back to an earlier statement, customers want solutions. It is cool to go away to training and get smothered in good food and perks. However, it’s  also known that sometimes that  goes to the manager, not  the person who’ll actually be doing the work! Also, training can’t solve certain types of problems.  There are many types of problems customers encounter, and they have different types of solutions. Videos may be better for things that occur infrequently, onboard help or job aids may meet other needs to unusual to be able to predict for training, etc. We don’t want to make customers happy, we want  to make them successful!
  8. We need ways to categorize people. It’s a natural human thing to categorize, including people. So if someone creates an appealing categorization that promises utility, hey that sounds like a good investment. Except, there are many problems! People aren’t easy to categorize, instruments struggle to be reliable, and vested interests will prey upon the unwary.  Anyone can create a categorization scheme, but validating it, and having it be useful, are both surprisingly big hurdles. Asking people questions about their behavior tends to be flawed for complex reasons. Using such tools for important decisions like hiring and tracking have proven to be unethical. Caveat emptor.
  9. Bandwagons are made to be jumped on. Face it, we’re always looking for new and better solutions. When someone links some new research to a better outcome, it’s exciting. There’s a problem, however. We often fall prey to arguments that appear to be new, but really aren’t. For instance, all the ‘neuro’ stuff unpacks to some pretty ordinary predictions we’ve had for yonks. Further, there are real benefits to machine learning and even artificial intelligence. Yet there’s also a lot of smoke to complement the sizzle. Don’t get misled. Do a skeptical analysis.  This holds doubly true for technology objects. It’s like a cargo cult, what’s has come down the pike must be a new gift from those magic technologists! Yet, this is really just another bandwagon. Sure, Augmented Reality and Virtual Reality have some real potential. They’re also being way overused. This is predictable, c.f. Powerpoint presentations in Second Life, but ideally is avoided. Instead, find the key affordances – what the technology uniquely provides – and match the capability to the need. Again, be skeptical.

My point here is that there can be misconceptions about learning  within  L&D, but it can also be outside perspectives that are flawed. So hopefully, I’ve now addressed both. I don’t claim that this is a necessary and complete set, just certain things that are worth noting. These are org learning myths that are worth trying to overcome, or so I think. I welcome your thoughts!

A message to CxOs: about L&D myths

by 3 Comments

If you’re a CEO, COO, CFO, and the like, are you holding L&D to account? Because much of what I see coming out of L&D doesn’t stand up to scrutiny. As I’ve cited in books and presentations, there’s evidence that L&D isn’t up to scratch. And I think you should know a few things that may be of interest to you. So here’re some L&D myths you might want to watch out for.

  1. If it looks like school, it must be learning. We’ve all been to school, so we know what learning looks like, right? Except, do you remember how effective school actually was? Did it give you many of the skills you apply in your job now?  Maybe reading and writing, but beyond that, what did you learn about business, leadership, etc? And how  did you learn those things? I’ll bet not by sitting and listening to lectures presented via bulletpoints. If it looks like schooling, it’s probably a waste of time and money. It should look more like lab, or studio.
  2. If we’re keeping our  efficiency in line with others, we’re doing good. This is a common belief amongst L&D: well, our [fill in the blank: employees served per L&D staff member | costs per hour of training | courses run per year | etc.] is the same or better than the industry average, so we’re doing good. No, this is all about efficiency, not effectiveness. If they’re not reporting on measurable changes in the improvement of business metrics, like sales, customer service, operations,e tc, they’re not demonstrating their worth. It’s a waste of money.
  3. We produce the courses our customers need. Can they justify that? It’s a frequent symptom that the courses that are asked for have little relation to the actual problem. There are many reasons for performance problems, and a reliable solution is to throw a course at it. Without knowing whether it’s truly a function of lack of skill. Courses can’t address problems like the wrong incentives, or a lack of resources. If you’re not ensuring that you’re only using courses when they make sense, you’re throwing away money.
  4. Job aids aren’t our job.  Performance should be the job, not just courses. As Joe Harless famously said: “Inside every fat course there‘s a thin job aid crying to get out.” There are many times when a job aid is a better solution than a course. To believe otherwise is one of the classic L&D myths. If they’re avoiding taking that on, they’re avoiding a cheaper and more effective solution.
  5. Informal learning isn’t our job. Well, it might not be if L&D truly doesn’t understand learning, but they should. When you’re doing trouble-shooting, research, design, etc., you don’t know the answer when you start. That’s learning too, and there is a role for active facilitation of best principles. Assuming people know how to do it isn’t justifiable. Informal learning is the key to innovation, and innovation is a necessary differentiation.
  6. Our LMS is all we need. Learning management systems (which is a misnomer, they’re course management systems) manage courses well. However, if they’re trying to also be resource portals, and social media systems, and collaboration tools, they’re unlikely to be good at all that. Yet those are also functions that affect optimal performance and continual innovation (the two things I argue  should be the remit of L&D). Further, you want the right tool for the job. One all-singing, all-dancing solution isn’t the way to bet for IT in general, and that holds true for L&D as well.
  7. Our investment in evaluation instruments is valuable. If you’re using some proprietary tools that purport to help you identify and characterize individuals, you’re probably being had. If you’re using it for hiring and promotion, you’re also probably violating ethical guidelines. Whether personality, or behavior, or any other criteria, most of these are methodologically and psychometrically flawed. You’re throwing away money. We have a natural instinct to categorize, but do it on individual performance, not on some flawed instrument.
  8. We have to jump on this latest concept.  There’re a slew of myths and misconceptions running around that are appealing and yet flawed. Generations, learning styles, attention spans, neuro-<whatever> and more are all appealing, and also misguided. Don’t spend resources on investing in them without knowing the real tradeoffs and outcomes.These are classic L&D myths.
  9. We  have to have this latest technology. Hopefully you’re resistant to new technologies unless you know what they truly will do for your organization. This holds true for L&D as well. They’re as prone to lust after VR and AR and AI as the rest of the organization. They’re also as likely to spend the money without knowing the real costs and consequences. Make sure they’re coming from a place where they know the unique value the technology brings!

There’s more, but that’s enough for now. Please, dig in. Ask the hard questions. Get L&D to be scrutable for real results, not platitudes. Ensure that you’re not succumbing to L&D myths. Your organization needs it, and it’s time to hold them to account as you do the rest of your organization. Thanks, and wishing you all the best.

Something that emerged from a walk, and, well, I had to get it off my chest. I welcome your thoughts.

Book hiccups

by Leave a Comment

As much as writing books is something I do (and I’m immodestly proud of the outcomes), they don’t always come out the way I expect. And that turns out to be true for almost every one!  So here, for the record and hopefully as both mea culpas and lessons learned, are my book hiccups. And you really don’t have to read this, unless you want some things to check for.

After my first book,  Engaging Learning, came out, someone asked me “how do I know it’s really your book?” He had a valid point, because while there was a bio, there was no picture of me. Somehow, I just expected it (and if memory serves, they’d asked for one). Yet it didn’t appear on the dust jacket nor on the author page. In fact, the only Wiley book that  did have my picture ended up being the next one.

Shortly after my next book came out,  Designing mLearning,  I got an email asking for clarification. The correspondent pointed to a particular diagram, and asked what I meant. It turns out, in editing (they’d outsourced it, I understand), someone had reversed the meaning of a caption for a diagram! Worse, I hadn’t caught it. At this time I can no longer find what it was, but it was an unhappy experience.

For my third book,  The Mobile Academy, I asked my friend and colleague John Ittelson to write the preface. And somehow, it wasn’t in the initial printing!  That was a sad oversight, but fortunately they remedied it very quickly.

I had been upset by how expensive the first two books were. Consequently, I was pleased to find out that my fourth,  Revolutionize  Learning & Development, that I really wanted to see do well, was priced much more reasonably. Of course, then I found out why; it was made with paper that wasn’t of the best quality. At least it’s affordable, and I continue to hear from people who have found it useful.

I’m happy to say that the next one,  Millennials, Goldfish & Other Training Misconceptions  has been hiccup free. After switching to ATD Press (they’d been a co-publisher of the previous book), they did a great job with the design, taking my notion of humorous sketches for each topic and executing against it graphically. It’s been well-recognized.

Unfortunately, as I just found out after getting my mitts on the most recent one,  Learning Science for Instructional Designers,  two of the four blurbs I solicited from esteemed colleagues don’t show up in the book!  They do show up on the ATD site, at least (and of course they’re on my own page for the book). I didn’t get a copy of the back cover beforehand, so I couldn’t have checked. My apologies to them. I checked, and it turns out having to do with a space issue because of book formatting. 🤷  Other than that, I’m  as  happy with this book as the last (that is, really happy)!

I can say that I’ve always tried to write in a way that focuses on the aspects that relate to our mental architecture. The goal is that as the technology changes, the implications are still appropriate. Our brains aren’t changing as fast at the tech! I guess I’m just not ready to accept planned obsolescence, so I’m keeping them available.

So there you have it, the book hiccups that can come with publishing. If you’ve made it this far, at least I hope you have some more things to check to make sure your books come out as good as possible.

 

Buzzwords and Branding

by Leave a Comment

I was reflecting on a few things on terminology, buzzwords and branding in particular. And, as usual, learning out loud, here are my reflections.

The script:

So I’ve been known to take a bit of a blade to buzzwords (c.f. microlearning). And, I reckon there’s a distinction between vocabulary and hype. Further, I get the need for branding (and have been slack on my own part).  So, here I talk about buzzwords and branding.

First, vocabulary is important. I’m a stickler (I’m sure some would say pedantic ;) about conceptual clarity. We need to have clear language to distinguish between different concepts. (You shouldn’t say ‘cat’ when you mean ‘dog’, someone’s likely to get a wee bit confused!)

And, to be clear, there’s internal and external vocabulary. For instance, other people don’t really care about objectives, they just want outcomes. This internal vocabulary can be shortcuts, and help us minimize what we need to say to still communicate. Brevity is the soul of wit, after all.

And then there’s hype. The distinction, I reckon, is when we start tossing in buzzwords that are new, drawn from elsewhere, and promise great things. Adaptive and neuro- are two examples of buzzphrases that are open to interpretation but sound intriguing. Yet they require careful examination.

Then, there’s branding. You attach a label to something to identify it specifically. Harold Jarche’s Personal Knowledge Mastery (PKM), for instance, is a brand for a framework. So, too, would be Michael Allen’s SAM (Successive Approximation Model) and CCAF (Context-Challenge-Activity-Feedback). They’re ways to package up good ideas. And of course, t0 take ownership.

This latter step, I confess, I’ve failed on. The alignment in Engaging Learning and the different categories of mobile are two places I dropped the ball. I recently tried a brief attempt to remedy another, when I released the Performance Ecosystem Maturity Model.

I  do have the 4C’s of Mobile, but while that turns out to be useful, it’s not the most important characterization. In a conversation with someone the other day, he asked what I called the mobile framework I mentioned and he found useful. And I didn’t have an answer. I’ve talked about it before, but I didn’t label it. And yet it’s kind of the most important way to look at mobile! I use it as the organizing framework when I talk about mobile (really, the performance ecosystem):

  • Augmenting formal learning
  • Performance support (mobile’s natural niche)
  • Social (more the informal)
  • Contextual (mobile’s unique opportunity)

I wasn’t sure what to brand this, so for the moment it’s the Four Modes of mLearning (4M? 4MM?).

And for games, that alignment I mentioned I briefly termed the EEA: Effectiveness-Engagement Alignment. The point is that the elements that lead to effective education practice, and the ones that lead to engaging experiences, have a perfect alignment. It’s been a good basis for design for me. But, again, that labeling came more than a decade after the book first came out.

Ok, so I was counting on the ‘Quinnovation’ branding. And that’s worked, but it’s not quite enough to hang products on. So…I’m working on it. (And it may be that having ‘Learnlets’ separate from Quinnovation is another self-inflicted impediment!)

Still, I think it’s important to distinguish between buzzwords and branding. And they shouldn’t be the same (trademarking ‘microlearning’, anyone ;). Again, vocabulary is important, for clarity, not hype. And branding is good for attribution. But they’re not the same thing. Those are my thoughts, what are yours?

Update on my workshops

by Leave a Comment

Just as I did an update on my books, it’s time to also let you know about some workshop opportunities. Together, I think they create a coherent whole. They’re scattered around a bit, so here I lay out how they fit together, how they’re run, what they cover, and how you can find them. They’re not free, but they’re reasonably priced, with reputable organizations. So here’s an update on my workshops.

First, they’re three pieces of the picture. I talk about two things, generally. It comes from my cheeky quip that L&D isn’t doing near what it could and should, and what it  is doing, it’s doing badly. So, that first part is about the larger performance ecosystem, and the second part is about learning experience design (LXD). And, that latter part actually pulls apart into two pieces.

I see LXD as the elegant integration of learning science with engagement. Thus, you need to understand learning science (and the associated elements). Then, you  also  need to understand what makes an engaging experience. So, two workshops address each of these.

The learning science workshop is being run under the auspices of HR.com (brokered through the Allen Academy). It’s under their professional education series, called Effective Learning Strategies. It’s a five week course (with a delayed sixth week). There are readings, a weekly session, and assignments. You can earn a certificate. In it I cover the basics of cognitive science, the learning outcomes, social/cultural/emotional elements, and the implications for design. It’s just what you need to know, and very much aligned with my forthcoming book!

The second part of the story is about the engagement side. While I’ve tried to boil down learning science into the necessary core, there are other resources. This isn’t well covered. And note, I’m  not talking about tarted-up drill-and-kill, gamification, ‘click to see more’, etc. Instead, I’m going deep into building, and maintaining: motivation, reducing anxiety, and more. Formally, it’s the Make It Meaningful workshop. This is a four week course, with videos to present the information, then live sessions to practice application, and takeaway assignments from the Learning Development Accelerator. It’s based upon the learnings from my book on designing learning games,  Engaging Learning,  but I’ve spent months this past summer making it more general, going deeper, validating the newest information, and making it accessible and comprehensible.

The final story is the performance ecosystem workshop. In what may seem a silly approach, it manifests as a course on mobile! However, once you recognize that mobile is about pretty much everything but courses (and can do contextual, which is an important new direction). It makes sense. When I was writing the mobile book, the intent was that it be a stealth approach to shift the L&D mindset away from just courses. Which, of course, was made more clear with my Revolutionize L&D book. So I hope you can see that this course, too, has a solid foundation. It’s about courses, performance support, informal and social learning, contextual opportunities, and strategy, in six weeks of online sessions, with a tiny bit of reading, and interim assignments. It’s by the Allen Academy directly.

Together, I think these three workshops provide the knowledge foundations you need to run a L&D operation. Two talk about what makes courses that are optimally engaging and effective, and one looks at the rest of the picture. Evidence suggests there’s a need. And I’ve worked hard to ensure that they’ve got the right stuff. So that’s an update on my workshops. I welcome your thoughts and feedback.  (And, yes, I’d like to pull them all together in one place, but I haven’t found a platform I like yet; stay tuned!)

 

Separate content from description

by 3 Comments

Once again facing folks who aren’t using styles, I was triggered to think more deeply about the underlying principle. That is, to separate content from description. It’s a step forward in what we can do with systems to bring about a more powerful human-aligned system.

And, as always, here’s the text, in case you (like me) prefer to read ;).

I‘ve ranted before about styles, but I want to make a slightly different pitch today. It‘s not just about styles, it‘s about the thinking behind it. The point is to separate content from description.

So, the point about styles is that they‘re a definition of formatting. You have elements of documents like headings at various levels, and body text, and special paragraphs like quotes, and so on. Then you have features, like font size, bolding and italics, color, etc. And what you see, too often, is people hand-formatting documents, choosing to do headers by increasing the font size, bolding, etc. And, importantly, having to go through and change them all manually if there‘s a desire for a change in look.  

The point of styles is instead merely to say this is a heading 1, this is a figure, this is a caption, and so on. Then, you separately say: heading 1s will be font size 16 bold and left-justified. Figures and captions will be centered, in font size 12. And so on. Then, should someone want to change how the document‘s formatted, you just change the definition of heading 1, and all the heading ones change.  

It goes further. You can define that all heading ones have a page break before (e.g. a new chapter in a book). And you can define new styles, like for a callout box (e.g. colored background), etc. You can have different heading ones for a book than for a white paper. And some styles can be based on others. So your headings can use the same font as your body text, and if you want it all to change, you change the source and the rest will change.  

Which is wonderful for writing, but the concept behind this is what‘s really important to get your head around. That is, separating out role from description. That‘s what‘s led me to be keen on content systems. The notion of pulling up content by description instead of hardwiring together content into an experience is the dream.  

It‘s all about beginning to use semantics, that is the meaning of things, as a manipulable tool. Many years ago, I led a project creating an adaptive learning system. We were going to have content objects defined by topic, and learning role, and tagged in terms of media, difficulty, and more. So you could   say: “pull a video on an example of diversity set in a sales office”. Our goal, with a suite of rules about what when to move up or back in difficulty, was to specify what learning content the learner should see next, etc.  

This is how adaptive platforms work. When Amazon or Netflix make a recommendation for you, there‘s not someone watching your behavior, instead it‘s a set of rules matching your particular actions to content recommendations. If you‘ve ordered a lot of British mysteries, and you haven‘t seen a particular series that lots of other people like, it‘ll be likely to be offered to you.  

This is the opportunity of the future. We can start doing this with learning (and coaching)!   We can start pulling together your learning goals, job role, current progress, current location (in time and space), etc, and offer you particular things that are appropriate for you. And, like our learning system, it might be recommendations of content, but you can choose others, or ignore, or…As Wayne Hodgins used to say, present the ‘right stuff‘: the right content to the right person at the right time in the right way….

The point being, just like styles, if we stop hardwiring things together, hand-formatting learning experiences, we can start offering personalized and even adaptive learning. Yes, there are technical backend issues, and more rigor in development, but this is the direction we can, and should, go. At least, that‘s my proposal, what say you?

Measuring Impact (or not)

by 6 Comments

So I saw a twitter thread pointing to an argument about how ROI is dead. And, well, that’s largely okay with me. However, the trigger for the post was from the results of Chief Learning Officer 2020 State of Learning report.  And, when I saw them, I saw some problems. The question is whether we’re measuring impact, or not. I’d like to go through them and evaluate each.

(Back to my usual prose, as I need visual support for this. ;)

So, in the report, they indicated that the respondents indicated the demonstrated impact of training in these ways:

  • General training output data
  • Training output data aligned with corporate initiatives
  • Learner satisfaction with training
  • Employee satisfaction with training availability
  • Employee engagement
  • Business impact
  • Employee performance data
  • Planned to actual budget, expense, revenue data for training group
  • Stakeholder satisfaction with training data
  • ROI measures
  • Net promoter score
  • Employee satisfaction with company data

Yikes!  Some of these are problematic at best.  Let’s look at why some of these might not be good measures of impact. And, let’s be clear; impact should be about positively affecting the organization in a meaningful way. Moving needles like fewer errors, more revenue, reduced costs, happier employees and customers, etc.

So, first, what  is  general training output data? If it’s like what I saw in (then) ASTD’s State of the Industry report, it’s metrics like employees served per L&D employee or cost/seat/hour for training. Which might a useful measure of efficiency, if you can come up with a principled basis for what a good number would be, and then see if you’re above or below that. Unfortunately, what people do is just compare themselves to the industry average. Is that a good indicator? How do you know? Do you want to be just ‘better than average’?

Then, training output data  aligned with corporate  initiatives.  Again, hard to say what this means (and I can’t seem to find the report). However, it sounds like it’s still efficiency, just doing that for things the business thinks are important.

And we go worse: learner satisfaction with training? Er, research I’ve read and heard cited (I think it’s from Salas, et al, but memory fails)  says that’s not valid. There’s a .09 correlation between what learners think of learning impact, and it’s actual impact. That’s zero with a rounding error. That’s all about making learning ‘fun’ (instead of ‘hard fun’). Yes, you do want them to think it’s  also been a good experience if you’re focusing on LXD, but that’s secondary.

Similarly, with satisfaction with training availability. What’s that matter? That’s not  impact!

Some good things buried here: employee engagement should be good; more engaged employees is a good thing. As long as it’s not at the cost of something else, like, say, impact?  And business impact is obviously good, as is employee performance data. Presumably positive business impact, and employee performance improving.

Planned to … stuff is all about efficiency again. And that’s ok, but only  after impact. Otherwise, well, we’re not  costing too much…?!?

Satisfaction again not good.

And, to the original point of the article. ROI?  Yes, what it costs you to move a needle should be less than the cost of what the needle was costing you.  However, I could be doing things that return the biggest ROI without doing the most important things. They can be different (e.g. a small program with a better ROI but less overall impact). So it’s only secondary.

Finally, employee satisfaction with company data? I have no idea what that means? But, again, ‘satisfaction’ isn’t really meaningful unless it’s based on real impact.

I’ve complained before about L&D measurement. Here it is, right in front of us. The answer to the question of whether we’re measuring impact or not appears to be ‘mostly not’.  We’re still (largely) measuring the wrong things. And we wonder why we don’t have credibility. Please, please, start designing to improve measurable gaps, and then actually improve the outcome. Otherwise, you’ve no idea whether that bum in that seat for an hour is doing the organization any good, versus just not costing too much relative to the industry average.

AI and Meaningful Practice

by 3 Comments

Again, a video of an idea I want to talk about. This time about AI and Meaningful Practice (just around 2 minutes). I welcome your thoughts.

By the way, I’m experimenting with video as a blog mechanism. A colleague mentioned that no one remembers the author of an article or post, but they do remember the speaker in a video. And much as I hate to do it, I do want people to associate my ideas with me! I welcome, very much, your feedback on this too!

Script:

Hi, I‘m Clark Quinn, Executive Director of Quinnovation (my vehicle for learning experience design strategy).    Today I want to talk about an insight I had, sparked in a conversation I had with a colleague.  We were talking about learning (of course), and the difference between knowledge versus practice.

I was reminded that we‘re now seeing AI technologies that can parse content and then answer questions about it.  We even see ones that can ask you questions about the content!  Which is part of learning.  But not all.

My realization was that, increasingly, these systems will take over this form of content presentation.  That is, we‘ll write a white paper, and an AI will parse it, then present it, and drill it.  Which is, after all, way too much of corporate learning.  See, for instance, the Serious eLearning Manifesto as a response.

Now, I‘ve always maintained that such systems aren‘t sufficient for real learning.  Meaningful learning includes more: motivation and contextualized practice.  Content presentation and quiz questions may be necessary, but by no means are they sufficient.  And that for now and the foreseeable future, AI will not be able to create those elements.  

This is the job of LXD: integrating learning science and deep engagement into experiences that transform us.  Which means that what L&D needs to do is stop doing information dump and knowledge test,  and learn how to do real learning experience design!  That, I suggest, is a noble pursuit (and, to be fair, what we should have been doing all along).  

Of course, there‘s also the necessary new role, per my last post/video, of being a facilitator of informal learning.  Coupling the optimal execution with continual innovation.  But, for now, I‘m suggesting we truly have to master everything that makes learning work, in particular meaningful practice.

That‘s my take, I welcome hearing yours.  Thanks for watching!