Learnlets

Secondary

Clark Quinn’s Learnings about Learning

Separate content from description

by 3 Comments

Once again facing folks who aren’t using styles, I was triggered to think more deeply about the underlying principle. That is, to separate content from description. It’s a step forward in what we can do with systems to bring about a more powerful human-aligned system.

And, as always, here’s the text, in case you (like me) prefer to read ;).

I‘ve ranted before about styles, but I want to make a slightly different pitch today. It‘s not just about styles, it‘s about the thinking behind it. The point is to separate content from description.

So, the point about styles is that they‘re a definition of formatting. You have elements of documents like headings at various levels, and body text, and special paragraphs like quotes, and so on. Then you have features, like font size, bolding and italics, color, etc. And what you see, too often, is people hand-formatting documents, choosing to do headers by increasing the font size, bolding, etc. And, importantly, having to go through and change them all manually if there‘s a desire for a change in look.  

The point of styles is instead merely to say this is a heading 1, this is a figure, this is a caption, and so on. Then, you separately say: heading 1s will be font size 16 bold and left-justified. Figures and captions will be centered, in font size 12. And so on. Then, should someone want to change how the document‘s formatted, you just change the definition of heading 1, and all the heading ones change.  

It goes further. You can define that all heading ones have a page break before (e.g. a new chapter in a book). And you can define new styles, like for a callout box (e.g. colored background), etc. You can have different heading ones for a book than for a white paper. And some styles can be based on others. So your headings can use the same font as your body text, and if you want it all to change, you change the source and the rest will change.  

Which is wonderful for writing, but the concept behind this is what‘s really important to get your head around. That is, separating out role from description. That‘s what‘s led me to be keen on content systems. The notion of pulling up content by description instead of hardwiring together content into an experience is the dream.  

It‘s all about beginning to use semantics, that is the meaning of things, as a manipulable tool. Many years ago, I led a project creating an adaptive learning system. We were going to have content objects defined by topic, and learning role, and tagged in terms of media, difficulty, and more. So you could   say: “pull a video on an example of diversity set in a sales office”. Our goal, with a suite of rules about what when to move up or back in difficulty, was to specify what learning content the learner should see next, etc.  

This is how adaptive platforms work. When Amazon or Netflix make a recommendation for you, there‘s not someone watching your behavior, instead it‘s a set of rules matching your particular actions to content recommendations. If you‘ve ordered a lot of British mysteries, and you haven‘t seen a particular series that lots of other people like, it‘ll be likely to be offered to you.  

This is the opportunity of the future. We can start doing this with learning (and coaching)!   We can start pulling together your learning goals, job role, current progress, current location (in time and space), etc, and offer you particular things that are appropriate for you. And, like our learning system, it might be recommendations of content, but you can choose others, or ignore, or…As Wayne Hodgins used to say, present the ‘right stuff‘: the right content to the right person at the right time in the right way….

The point being, just like styles, if we stop hardwiring things together, hand-formatting learning experiences, we can start offering personalized and even adaptive learning. Yes, there are technical backend issues, and more rigor in development, but this is the direction we can, and should, go. At least, that‘s my proposal, what say you?

Measuring Impact (or not)

by 6 Comments

So I saw a twitter thread pointing to an argument about how ROI is dead. And, well, that’s largely okay with me. However, the trigger for the post was from the results of Chief Learning Officer 2020 State of Learning report.  And, when I saw them, I saw some problems. The question is whether we’re measuring impact, or not. I’d like to go through them and evaluate each.

(Back to my usual prose, as I need visual support for this. ;)

So, in the report, they indicated that the respondents indicated the demonstrated impact of training in these ways:

  • General training output data
  • Training output data aligned with corporate initiatives
  • Learner satisfaction with training
  • Employee satisfaction with training availability
  • Employee engagement
  • Business impact
  • Employee performance data
  • Planned to actual budget, expense, revenue data for training group
  • Stakeholder satisfaction with training data
  • ROI measures
  • Net promoter score
  • Employee satisfaction with company data

Yikes!  Some of these are problematic at best.  Let’s look at why some of these might not be good measures of impact. And, let’s be clear; impact should be about positively affecting the organization in a meaningful way. Moving needles like fewer errors, more revenue, reduced costs, happier employees and customers, etc.

So, first, what  is  general training output data? If it’s like what I saw in (then) ASTD’s State of the Industry report, it’s metrics like employees served per L&D employee or cost/seat/hour for training. Which might a useful measure of efficiency, if you can come up with a principled basis for what a good number would be, and then see if you’re above or below that. Unfortunately, what people do is just compare themselves to the industry average. Is that a good indicator? How do you know? Do you want to be just ‘better than average’?

Then, training output data  aligned with corporate  initiatives.  Again, hard to say what this means (and I can’t seem to find the report). However, it sounds like it’s still efficiency, just doing that for things the business thinks are important.

And we go worse: learner satisfaction with training? Er, research I’ve read and heard cited (I think it’s from Salas, et al, but memory fails)  says that’s not valid. There’s a .09 correlation between what learners think of learning impact, and it’s actual impact. That’s zero with a rounding error. That’s all about making learning ‘fun’ (instead of ‘hard fun’). Yes, you do want them to think it’s  also been a good experience if you’re focusing on LXD, but that’s secondary.

Similarly, with satisfaction with training availability. What’s that matter? That’s not  impact!

Some good things buried here: employee engagement should be good; more engaged employees is a good thing. As long as it’s not at the cost of something else, like, say, impact?  And business impact is obviously good, as is employee performance data. Presumably positive business impact, and employee performance improving.

Planned to … stuff is all about efficiency again. And that’s ok, but only  after impact. Otherwise, well, we’re not  costing too much…?!?

Satisfaction again not good.

And, to the original point of the article. ROI?  Yes, what it costs you to move a needle should be less than the cost of what the needle was costing you.  However, I could be doing things that return the biggest ROI without doing the most important things. They can be different (e.g. a small program with a better ROI but less overall impact). So it’s only secondary.

Finally, employee satisfaction with company data? I have no idea what that means? But, again, ‘satisfaction’ isn’t really meaningful unless it’s based on real impact.

I’ve complained before about L&D measurement. Here it is, right in front of us. The answer to the question of whether we’re measuring impact or not appears to be ‘mostly not’.  We’re still (largely) measuring the wrong things. And we wonder why we don’t have credibility. Please, please, start designing to improve measurable gaps, and then actually improve the outcome. Otherwise, you’ve no idea whether that bum in that seat for an hour is doing the organization any good, versus just not costing too much relative to the industry average.

AI and Meaningful Practice

by 3 Comments

Again, a video of an idea I want to talk about. This time about AI and Meaningful Practice (just around 2 minutes). I welcome your thoughts.

By the way, I’m experimenting with video as a blog mechanism. A colleague mentioned that no one remembers the author of an article or post, but they do remember the speaker in a video. And much as I hate to do it, I do want people to associate my ideas with me! I welcome, very much, your feedback on this too!

Script:

Hi, I‘m Clark Quinn, Executive Director of Quinnovation (my vehicle for learning experience design strategy).    Today I want to talk about an insight I had, sparked in a conversation I had with a colleague.  We were talking about learning (of course), and the difference between knowledge versus practice.

I was reminded that we‘re now seeing AI technologies that can parse content and then answer questions about it.  We even see ones that can ask you questions about the content!  Which is part of learning.  But not all.

My realization was that, increasingly, these systems will take over this form of content presentation.  That is, we‘ll write a white paper, and an AI will parse it, then present it, and drill it.  Which is, after all, way too much of corporate learning.  See, for instance, the Serious eLearning Manifesto as a response.

Now, I‘ve always maintained that such systems aren‘t sufficient for real learning.  Meaningful learning includes more: motivation and contextualized practice.  Content presentation and quiz questions may be necessary, but by no means are they sufficient.  And that for now and the foreseeable future, AI will not be able to create those elements.  

This is the job of LXD: integrating learning science and deep engagement into experiences that transform us.  Which means that what L&D needs to do is stop doing information dump and knowledge test,  and learn how to do real learning experience design!  That, I suggest, is a noble pursuit (and, to be fair, what we should have been doing all along).  

Of course, there‘s also the necessary new role, per my last post/video, of being a facilitator of informal learning.  Coupling the optimal execution with continual innovation.  But, for now, I‘m suggesting we truly have to master everything that makes learning work, in particular meaningful practice.

That‘s my take, I welcome hearing yours.  Thanks for watching!

 

Flow, workflow, and learning

by 3 Comments

On LinkedIn, a colleague asked “Why do people think that integrating content in the flow of work equals learning in the flow of work?” An apt question. My (flip) response was “because marketing”. And I think there’s a lot to that. But, a comment prompted me to think a little bit deeper, because ‘flow’ is its own meaningful concept and we need to be careful about meaning. So here are some reflections on flow, workflow, and learning.

The response that triggered my reflection was:

I can’t recall the last time I told someone that I was in the “flow” of work today and learned so much!!

Flow state(Which is pretty funny!) The comment was a bit pointed, but it made me think about being in the ‘flow’ state, and the relationship with learning. I’ve previously pointed out how Csikszentmihalyi’s Flow and Vygotsky’s Zone of Proximal Development  (ZoPD) are essentially the same. If the difficulty is too far above your skill level, the experience is frustrating. If it’s too easy, it’s boring. And in between is the flow state, and where learning happens.

Now, when we’re in the ‘flow’ at work (which is different than being in the workflow), we’re performing optimally. And I’m not sure learning happens there. Similarly with the ZoPD. You’re working and I’m not sure learning happens  then. When I state that learning is action and reflection, I think reflection is a necessary component.

Now, the original complaint talked about learning in the workflow, and opined that content in the workflow won’t necessarily equal learning. Another comment pointed out what I believe is often conflated with “workflow learning”, and that’s performance support. There are lots of reasons that we might want content in the workflow to help us succeed, but it may have nothing to do with learning. If, indeed, learning is to happen, it might need some content, and feedback, and so actually break the flow!

Now, I also recognize that many times we’re in the flow of work, but not in the ‘flow’  zone. So, we could definitely be learning in the workflow. And it happens by deciding to look up the answer to some contextually relevant question. Or from a comment from a person. But it’s a bit different than being in the zone, and we’d like to be there in our work too!

And, I wonder whether Vygostky’s ZoPD really aligns with the Flow Zone, or if it needs to be coupled with some offline reflection. It’s certainly possible. Maybe the flow zone is a superset of the ZoPD. More to ponder.

There isn’t a real revelation here about flow, workflow, and learning, other than we have to keep our concepts straight. We need to recognize when we’re supporting performance, and when we’re learning. And we need to be clear about workflow, and being in the flow zone. And there may be more here to unpack. Thoughts?

 

In Defense of Science

by 3 Comments

I previously wrote in defense of cog psych. Here, I want to go broader. Not my usual topic, but… I feel the need to rail in defense of science.

When I’m talking about science, I’m talking about the systematic exploration of how things work. It’s about theorizing, and testing, and refining. It’s about having rigor in that, being systematic and principled. And, importantly, it’s about sharing the results and building on the works of others. Interestingly, it’s  not a human universal, but instead emerged relatively recently. However, there’s a reason it’s been so successful.

And, let’s be honest, science has flaws. There have been recent problems in replicability. Another problem is academic politics; new ideas can struggle to get recognized. Pressure to publish can lead to fake data. Folks with money can influence what research gets done, and the outcomes. Similarly, politics can play a role. Like democracy, it’s not perfect, but…and this is an important  but…there’s nothing better.

The evidence for science are the things that we’ve come to count on: sanitation, transportation, medicine, the list goes on. Your ability to read this depends on science. Most of what we use all day every day has been improved by science. So, too, some bad things, like how successful marketing can be at snagging your attention. Yes, we need to use it wisely. Science can help there, too! And, importantly, most scientists are ethical, caring, diligent individuals. They do what they do  for science, not for wealth, not for fame (except amongst their colleagues, which goes back to doing it  for science), and most certainly not to support conspiracies.

So, trying to pick and choose what science to believe isn’t a great bet. Unless you have a deep background in a particular domain, trying to ascertain the validity is challenging. You may listen to disparate voices, but not if they’re flying the face of a concerted viewpoint of people who have spent the requisite time to be true experts. In my mind, you’re either for science, or not. Saying “well, I’m not for  this science because someone said it’s controversial”, then, is just not on.

Yes, there are controversies around most science:  that’s how it advances. But there’s also essential truths that most every reputable scientist in the field will agree to. And that’s how we build products, services, and ultimately societies.  I was trained as a scientist (though I’m more of an engineer, tracking it and applying it to solve real problems). I know, in my field, what makes sense and what’s silly. And then, in other fields, I look to what the received wisdom is. And I know what sorts of people to listen to, and it’s not politicians, or pundits. Unless they listen to science.  The best guidance comes from the folks who know the field in question. And that holds true for medicine as well as meteorology.

And sure, I too could wish I lived in a world where magic worked. But if you think about it, they, too, use systematic experimentation to find out what works. Whether Earthsea or Hogwarts, they go to schools to learn and there the professors are studying. But here, magical thinking doesn’t work. Science is what has let us knock back polio, generate electricity from sunlight, and walk on the moon.

So, if you do want to go against what the scientists or reliable interpreters tell you, don’t do it piecemeal. Abandon all the science, because you’re unlikely to get it right in a domain that’s not your expertise. If  anyone is telling you contrary to what’s known, question their motives! People mislead for lots of reasons, from money to mischief. If you let them, they’ll hurt you in ways that may be stark or subtle.

If they’re steering you away from something that has been shown to be better than the alternative, you should be wary. Their tricks are myriad: lack of context, distortions, selected subsets, and outright lying. For instance, our brains are wired to see patterns. If we’re pointed to them, we’ll see them. We’re also biased to look for evidence that confirms our beliefs, and avoid what contradicts it. Thus, it’s easy to gin up potential conspiracies, despite the incredible challenge in actually pulling them off!

I’m putting it out there. I can say that, in defense of science, it’s better than any other approach. That’s my stance, what’s yours?

 

Ritual

by 2 Comments

I’ve talked before about the power of ritual, but while powerful, it also seemed piecemeal. That is, there were lots of hints, but not a coherent theory. That has now changed. I recently found a paper by Nicholas Hobson & colleagues (Schroeder, Risen, Xylagatas, & Inzlicht; warning, PDF) titled  The Psychology of Rituals  that creates an integrated framework. And while my take simplifies it down, I found it interesting.

At core, what the model suggests is that there are two components that are linked together. The first element are things that involve the senses. The second element are the semantics we’re looking to create allegiance and adherence too. And there are important elements about this relationship.

There are a number of elements that are on tap for involving the senses. Certain movements, sounds, and words said or to be spoken can be used. There can also be food, drink, smells, and more. Objects also. Timing is an element; at the micro level of things in order, and at the macro level of the triggers for the ritual.

Semantics come, of course, from your needs. It can be about things you want people to believe, or a set of values you want people to subscribe to. Or, of course, both. From the design purpose, I’d suggest it’s about agreeing to be a member of a community of practice; to undertake certain actions when appropriate, and to uphold certain values.

Interestingly, according to their model, the relationship between the two is effectively arbitrary. That is, there is no intrinsic relationship between what you’re signifying, and how you do so. Rituals are about the practices. Which means you could in theory do just about  anything to make the relationship.

The other thing is that the ritual has to be invariable in its aspects. You define it, and so do it. Note that the execution can vary considerably; from several times a day to upon certain triggering conditions. So, for instance, having completed a course, or before engaging in certain activities.

While such a definition gives us lots of freedom, it also doesn’t necessarily serve as a guide for design. Still, thinking about it in this way does suggest the utility in developing deeply held beliefs and appropriately practiced behaviors. At least, that’s how I see it. You?

What is wrong with (higher) education?

by 7 Comments

I was having a conversation with a colleague, sparked by dropping enrollments in unis. Not surprisingly, we ended up talking about flaws in higher education. He suggested that they don’t get it, and I agreed. He was thinking that they get the tech, but not the learning. I think it’s more complex. There are those that get some parts of the learning right. Just not enough, and not all of it right. Thinking further, post-convo, it occurs to me that there is a layer beneath the surface that matters. So I want to consider what is wrong with higher education.

And, let’s be clear, I’m  not talking about the problems with tuition and administration. Yes, tuition’s risen faster than the cost of living. And yes, there’s little commercial pressure to keep universities free from the persistent creep of increasing administration. I saw an interesting article talking about how universities without a solid financial foundation,  and ones without a good value proposition, will perish. It’s the latter I’m talking about.

I previously mentioned the three pillars I think create a valid learning offer:

  • a  killer learning experience,
  • being a partner in your success
  • and developing you as an individual.

I suggest that all three are doable, but it occurred to me that there’s a bit more to unpack.

The ‘being a partner in your success’ bit is most frequently seen. Here, it’s about looking for signs of trouble and being proactive about reaching out and assisting. It’s not ‘sink or swim’, but recognizing there can be troubles and helping learners cope. The Predictive Analytics work that Ellen Wagner did is the type of opportunity we have here.

The ‘developing you as an individual’ is really building your more general skills: communicating, working with others, a positive attitude, knowing how to search, etc. And, of course, knowing how to continue to learn. Given the rate of change, most of what you learn as the core of a degree may well be out of date in short order!  But you can’t address these skills on their own, they’re specifically about how you do domain things.  And that’s a layer I’ve yet to see.

And the ‘killer learning experience’ is a second area where I think folks still aren’t doing well. My short (and admittedly cheeky) statement about education is that they’re wrong on two things, the curriculum and pedagogy, other than that they’re fine. Most universities aren’t doing a good job of curriculum, focusing on knowledge instead of skills. And some are moving in a good direction. Startups are addressing this area as well.

The other problem is the pedagogy. There’re two elements here: the learning design, and engagement. Too often, it’s still the ‘information dump and knowledge test’. But even when that’s right, making it truly meaningful for the learners is sadly neglected. Even professors who care often forget to put the ‘why’ into the syllabus.

In short, what is wrong with higher education is the ability to successfully execute on  all these points. (It’s true for other education, too, but…) I’ve seen efforts that address one, or two (and plenty that get none right). However, as of yet, I have not seen anyone doing it the way it could be done.  It’s doable, but not without some serious attention to not only the elements, but their successful integration. And it’s important enough that we should be doing it. At least, that’s what I think. So, what do  you think?

Personalized and adaptive learning

by 1 Comment

For reasons that are unclear even to me, I was thinking about personalized versus adaptive learning. They’re similar in some ways, but also different. And a way to distinguish them occurred to me. It’s kinda simplistic, but I think it may help to differentiate personalized and adaptive learning.

As background, I led a project to build an intelligently adaptive learning platform. We were going to profile learners, but then also track their ongoing behavior. And, on this basis, we’d serve up something appropriate for learner X versus learner Y. (We’d actually recommend something, and they could make other choices.)

It was quite the research endeavor, actually, as the CEO had been inspired by Guilford’s learning model. I dug into that and all the learning styles literature, and cognitive factor analysis, and content models around learning objectives, and revisited my interest in intelligent tutoring, and more. I was able to hire a stellar team, and create an approach that was scientifically scrutable (e.g. no learning ‘styles’ :). We got it up and running before, well, 2001 happened and the Internet bubble burst and…

In some sense, the system was really both, in the way I’m thinking about it. I’ve seen different definitions, and one has adaptive as a subset of personalized, but I’m going a different way. I think of personalized as pre-planned alternatives for different groups, whereas adaptive reacts to the learner’s behavior.

Our use of initial profiling, if we only used that, would be personalized. The ongoing adaptation is what made it adaptive. We had rules that would prioritize preferences, but we’d also use behavior to update the learner model. It’s something they’re doing now, but we had it a couple of decades ago.

So, my simple way of thinking about personalized versus adaptive is that personalized is based upon who you are: your role, largely. We’d swap out examples on marketing for people selling services versus those selling products, for instance. Or if we’re talking negotiation, a vendor might get a different model than a lawyer.

Adaptive, on the other hand, is based upon what you  do. So, for instance, if you did poorly on the last problem, we might not give you a more difficult one, but give you another at the same level. Twice in a row doing badly, we might bring you another example, or even revisit the concept. This is what intelligent tutoring systems do, they just tend to require a rigorous model of expertise.

Of course, you could get more complicated. Personalization might have a more and less supportive path, depending on your anxiety and confidence. Similarly, adaptive might throw in an encouraging remark while showing some remedial materials.

At any rate, that’s how I differentiate personalized and adaptive learning. Personalized is pre-set based upon some determined differences that suggest different learning paths. Adaptive calculates on the fly and changes what the learner sees next.  How do you see it?

Skills, competencies, and moving forward

by 3 Comments

I was asked, recently, about skills versus competencies. The context was an individual who saw orgs having competency frameworks, but only focusing on skill development. The question was where the focus should be. And I admit I had to look up the difference first! But then I could see where the emphasis should be on skills, competencies, and moving forward.

Now, the reason I joined with IBSTPI (the International Board for Standards in Training, Performance, and Instruction) was to learn more about competencies. So I didn’t feel inadequate looking it up (and probably should’ve asked my colleagues), but my search revealed a consistent viewpoint that kept me from having to bother them. The story was that there are individual skills, but that it takes more to do a job.

Competencies are suites of skills, knowledge, and attitudes* that create the ability to apply them in context to accomplish goals.  So you may be able to address customer objections, but there’s more to closing a sale than that. Competencies are aggregates of skills; they’re not just focused on what, but how. They’re a richer picture, based upon performance.

Should you care? It seems to me that you should. The clear implication is that if you only focus on skills, you may be missing other elements. You could develop skills and still not develop the ability to succeed. Thus, organizations are increasingly needing to focus on contextualized abilities to perform.

I’ll go further. In the days of optimizing performance, skills could potentially be sufficient. You knew what you had to do, and you had to do it. However, increasingly optimal execution is only the cost of entry, and continual innovation is the only sustainable differentiator. And that, I suggest, comes from competencies beyond skills.

Increasingly, you see orgs moving to competency-based hiring as well as development. Performance management likewise benefits from focusing on competencies.

Overall, my take is that when you’re looking at skills, competencies, and moving forward, competencies offer more power.

*”attitude” added based upon sound critique from Paul Kirschner.

Addressing fear in learning

by 4 Comments

One of my mantras in ‘make it meaningful‘ is that there’re three things to do. And one of those  was kind of a toss away, until a comment in a conversation with a colleague brought it home. So here’s a first take at addressing fear in learning.

The mantra, to be clear, is that you have 3 major hurdles to overcome in getting someone to ‘buy in’ to learning:

  • I  do need this
  • I don’t know it already
  • and I trust this experience will address that

I’ve focused mostly on the first, to date. The second is for the case where someone’s overconfident in their own abilities. However, the latter one was a toss-away, until…

My colleague mentioned how in trying to train data analysis, you could be coming up against decades of a belief such as ‘I don’t do well at math’. And I saw how you could have anxiety or a lack of confidence that this learning could address it.

Which makes it clear that you need to know the audience, and anticipate barriers. How can you address such a situation? I think you have to make sure that you make it steady and slow enough, or that it’s misperceived. So here, I could see either suggesting “we’ll take it slow enough and make it simple enough that you’ll find it easy” or “you may think data analytics is about math, but that’s the least part of it, it’s really about asking and answering questions”.

The point being, you need that trust, and that means addressing any barriers. It’s addressable, but you need to be aware. I also wonder if the typical elearning experience might have undermined trust such that there would have to be a series of successes to reestablish the trust that a learning unit  should have. However, if we start regularly addressing all three, we have a start. That includes establishing the need, removing false conceptions, and addressing fear in learning. Those are my thoughts, what are yours?