Quinnovating

Update on my events

17 June 2021 by Clark Leave a Comment

In January I posted about my upcoming webinars (now past), workshops, etc. As things open up again (yay, vaccines), some upcoming events will be happening live! And, of course, virtual. In fact, one starts next week! So I thought it time to update you on the things I’ll be doing. Then we’ll get back to my regular posts ;). So here’s an update on my events.

First, starting next week, is the Learning Development Conference, by the Learning Development Accelerator (caveat: I’m on their advisory board). Last year, it was an experiment. They did several things very well: it was focused on evidence-based approaches, it created timings that worked for a broad section of the world’s populace (e.g. live sessions were offered twice, once early once late), and it had asynchronous content as well as synchronous. It also had ways to maintain contact and discussions. As a result, it was a success, leading to the Accelerator and this second event.

It’s for six weeks, and first I’ve got an asynchronous course on learning science (a subset of the bigger one I do as a blended workshop for HR.com/Allen Academy). I’m also doing two live sessions (at different times) on some of the new results from cognitive science. I’m already dobbed in for one debate, and they’ll likely call on me for more. There are also a suite of the top names in evidence-based L&D appearing doing either or both of live or asynchronous content.

Second, at the end of August, I’ll be speaking at ATD’s International Conference and Exposition. This is a live event in Salt Lake City. (My first since the pandemic!) Of course I’m speaking on learning science; the topic of my book with them. There could even be a book-signing event! If you don’t know ATD’s ICE, it’s huge, both a blessing and curse. Lots of quality content (ok, mostly ;), almost too many people to find your friends, but lots of new friends to make, with broad coverage. Also, a big exposition (maybe smaller this year ;).

Third, I’ll be at the Learning Guild’s DevLearn again this year. This has always been one of the best conferences because the Guild runs good events (caveat: I’m their first Guild Master). They want it to grow, of course, but as yet it’s still be reasonably sized, and with quality content. For one, I’ll be speaking on learning science implications.

I’ll also be running a pre-conference workshop on Making Learning Meaningful. And this is, I suggest, truly of interest. I’ve been seeing more and more examples of well-designed content that’s still lacking in engagement, and this workshop is all about that. It’s an area I’ve been actively exploring and synthesizing into practical implications. Like in the series I did on the topic here, I cover how to hook initial interest, then maintain it through the experience. Also considered are the implications for the elements of learning, and a process to make it practical.

I recommend all three (or I wouldn’t be inclined to speak at them). So that’s the current update on my events. Hope to see you at one or another!

Exploring Exploration

15 June 2021 by Clark Leave a Comment

Learning, I suggest, is action and reflection. (And instruction should be designed action and guided reflection.) What that action typically ends up being is some sort of exploration (aka experimentation). Thus, in my mind, exploration is a critical concept for learning. That makes it worth exploring exploration.

In learning, we must experiment (e.g. act) and observe and reflect on the outcomes. We learn to minimize surprise, but we also act to generate surprise. I stipulate that we do so when the costs of getting it wrong are low. That is, making learning safe. So providing a safe sandbox for exploration is a support for learning. Similarly, have low consequences for mistakes generated through informal learning.

However, our explorations aren’t necessarily efficient nor effective. Empirically, we can make ineffective choices such as changing more than one variable at a time, or missing an area of exploration completely. For instruction, then, we need support. Many years ago, Wallace Feurzig argued for guided exploration, as opposed to free search (the straw man used to discount constructivist approaches). So putting constraints on the task and/or the environment can support making exploration more effective.

Exploration also drives informal learning. Diversity on a team, properly managed, increases the likelihood of searching a broader space of solutions than otherwise. There are practices that increase the effectiveness of the search. Similarly, exploration should be focused on answering questions. We also want serendipity, but there should be guidelines that keep the consequences under control.

By making exploration safe and appropriately constrained, we can advance our understanding most rapidly, either helping some folks learn what others know, or advance what we all know. Exploration is a key to learning, and we need to understand it. Thus, we should also keep exploring exploration!

New recommended readings

8 June 2021 by Clark Leave a Comment

Of late, I’ve been reading quite a lot, and I’m finding some very interesting books. Not all have immediate take homes, but I want to introduce a few to you with some notes. Not all will be relevant, but all are interesting and even important. I’ll also update my list of recommended readings. So here are my new recommended readings. (With Amazon Associates links: support your friendly neighborhood consultants.)

First, of course, I have to point out my own Learning Science for Instructional Designers. A self-serving pitch confounded with an overload of self-importance? Let me explain. I am perhaps overly confident that it does what it says, but others have said nice things. I really did design it to be the absolute minimum reading that you need to have a scrutable foundation for your choices. Whether it succeeds is an open question, so check out some of what others are saying. As to self-serving, unless you write an absolute mass best-seller, the money you make off books is trivial. In my experience, you make more money giving it away to potential clients as a better business card than you do on sales. The typically few hundred dollars I get a year for each book aren’t going to solve my financial woes! Instead, it’s just part of my campaign to improve our practices.

So, the first book I want to recommend is Annie Murphy Paul’s The Extended Mind. She writes about new facets of cognition that open up a whole area for our understanding. Written by a journalist, it is compelling reading. Backed in science, it’s valuable as well. In the areas I know and have talked about, e.g. emergent and distributed cognition, she gets it right, which leads me to believe the rest is similarly spot on. (Also her previous track record; I mind-mapped her talk on learning myths at a Learning Solutions conference). Well-illustrated with examples and research, she covers embodied cognition, situated cognition, and socially distributed cognition, all important. Moreover, there’re solid implications for the redesign of instruction. I’ll be writing a full review later, but here’s an initial recommendation on an important and interesting read.

I’ll also alert you to Tania Luna’s and LeeAnn Renninger’s Surprise. This is an interesting and fun book that instead of focusing on learning effectiveness, looks at the engagement side. As their subtitle suggests, it’s about how to Embrace the Unpredictable and Engineer the Unexpected. While the first bit of that is useful personally, it’s the latter that provides lots of guidance about how to take our learning from events to experiences. Using solid research on what makes experiences memorable (hint: surprise!) and illustrative anecdotes, they point out systematic steps that can be used to improve outcomes. It’s going to affect my Make It Meaningful work!

Then, without too many direct implications, but intrinsically interesting is Lisa Feldman Barrett’s How Emotions Are Made. Recommended to me, this book is more for the cog sci groupie, but it does a couple of interesting things. First, it creates a more detailed yet still accessible explanation of the implications of Karl Friston’s Free Energy Theory. Barrett talks about how those predictions are working constantly and at many levels in a way that provides some insights. Second, she then uses that framework to debunk the existing models of emotions. The experiments with people recognizing facial expressions of emotion get explained in a way that makes clear that emotions are not the fundamental elements we think they are. Instead, emotions social constructs! Which undermines, BTW, all the facial recognition of emotion work.

I also was pointed to Tim Harford’s The Data Detective, and I do think it’s a well done work about how to interpret statistical claims. It didn’t grip me quite as viscerally as the afore-mentioned books, but I think that’s because I (over-)trust my background in data and statistics. It is a really well done read about some simple but useful rules for how to be a more careful reviewer of statistical claims. While focused on parsing the broader picture of societal claims (and social media hype), it is relevant to evaluating learning science as well.

I hope you find my new recommended readings of interest and value. Now, what are you recommending to me? (He says, with great trepidation. ;)

The case for model answers (and a rubric)

3 June 2021 by Clark 4 Comments

As I’ve been developing online workshops, I’ve been thinking more about the type of assessment I want. Previously, I made the case for gated submissions. Now I find another type of interaction I’d like to have. So here’s the case for model answers (and a rubric).

As context, many moons ago we developed a course on speaking to the media. This was based upon the excellent work of the principals of Media Skills, and was a case study in my Engaging Learning book. They had been running a face to face course, and rather than write a book, they wondered if something else could be done. I was part of a new media consortium, and was partnered with an experienced CD ROM developer to create an asynchronous elearning course.

Their workshop culminated in a live interview with a journalist. We couldn’t do that, but we wanted to prepare people to succeed at that as an optional extra next step. Given that this is something people really fear (apocryphally more than death), we needed a good approximation. Along with a steady series of exercises going from recognizing a good media quote, and compiling one, we wanted learners to have to respond live. How could we do this?

Fortunately, our tech guy came up with the idea of a programmable answering machine. Through a series of menus, you would drill down to someone asking you a question, and then record an answer. We had two levels: one where you knew the questions in advance, and the final test was one where you’d have a story and details, but you had to respond to unanticipated questions.

This was good practice, but how to provide feedback? Ultimately, we allowed learners to record their answers, then listen to their answers and a model answer. What I’d add now would be a rubric to compare your answer to the model answer, to support self-evaluation. (And, of course, we’d now do it digitally in the environment, not needing the machine.)

So that’s what I’m looking for again. I don’t need verbal answers, but I do want free-form responses, not multiple-choice. I want learners to be able to self-generate their own thoughts. That’s hard to auto-evaluate. Yes, we could do whatever the modern equivalent to Latent Semantic Analysis is, and train up a system to analyze and respond to their remarks. However, a) I’m doing this on my own, and b) we underestimate, and underuse, the power of learners to self-evaluate.

Thus, I’m positing a two stage experience. First, there’s a question that learners respond to. Ideally, paragraph size, though their response is likely to be longer than the model one; I tend to write densely (because I am). Then, they see their answer, a model answer, and a self-evaluation rubric.

I’ll suggest that there’s a particular benefit to learners’ self-evaluating. In the process (particularly with specific support in terms of a mnemonic or graphic model), learners can internalize the framework to guide their performance. Further, they can internalize using the framework and monitoring their application to become self-improving learners.

This is on top of providing the ability to respond in richer ways that picking an option out of those provided. It requires a freeform response, closer to what likely will be required after the learning experience. That’s similar to what I’m looking for from the gated response, but the latter expects peers and/or instructors to weigh in with feedback, where as here the learner is responsible for evaluating. That’s a more complex task, but also very worthwhile if carefully scaffolded.

Of course, it’d also be ideal if an instructor is monitoring the response to look for any patterns, but that’s outside the learners’ response. So that’s the case for model answers. So, what say you? And is that supported anywhere or in any way you know?

How to be an elearning expert

1 June 2021 by Clark 3 Comments

I was asked (and have been a time or two before): “What’s the one most important thing you’d like to tell to be successful Ed Tech industry leader?” Of course there wasn’t just one ;). Still, looking at colleagues who I think fit that characterization, I find some commonalities that are worth sharing. So here’s one take on how to be an elearning expert.

Let’s start with that ‘one thing’. Which is challenging, since it’s more than one thing! Still, I boiled it down into two components: know your stuff, and let people know. That really is the core. So let’s unpack that some more. The first thing is to establish credibility. Which means demonstrating that you track and promote the right stuff.

Some folks have created a model that they tout. Cathy Moore has Action Mapping, Harold Jarche has PKM, Con Gottfredson has the 5 moments of need, and so on. It’s good having a model, if it’s a good, useful one (there are people who push models that are hype or ill-conceived at best). Note that it’s not necessarily the case that these folks are just known for this model, and most of these folks can talk knowledgeably about much more, but ‘owning’ a model that is useful is a great place to be. (I occasionally regret that I haven’t done a good job of branding my models.) They understand their model and its contribution, it’s a useful one, and therefore they contribute validly that way and are rightly recognized.

Another approach like this is owning a particular domain. Whether gaming (e.g. Karl Kapp), visuals (Connie Malamed), design (Michael Allen), mixed realities (Ann Rollins), AI (Donald Clark), informal (Jane Hart), evaluation (Will Thalheimer), management (Matt Richter), and so on, they have deep experience and a great conceptual grasp in a particular area. Again, they can and do speak outside this area, but when they talk about these topics in particular, what they say is worthy of your attention!

Then there are other folks who don’t necessarily have a single model, but instead reliably represent good science. Julie Dirksen, Patti Shank, Jane Bozarth, Mirjam Neelen, and others have established a reputation for knowing the learning science and interpreting it in accurate, comprehensible, and useful ways.

The second point is that these folks write and talk about their models and/or approaches. They’re out there, communicating. It’s about reliably saying the important things again and again (always with a new twist). A reputation doesn’t just emerge whole-cloth, it’s built step by step. They also practice what they preach, and have done the work so they can talk about it. They talk the talk and walk the walk. Further, you can check what they say.

So how to start? There are two clear implications. Obviously, you have to Know. Your. Stuff! Know learning, know design, know engagement, know tech. Further, know what it means in practice! You can focus deeply in one area, or generate one useful and new model, or have a broad background, but it can’t just be in one thing. It’s not just all your health content for one provider. What you’re presenting needs to be representative and transferable. Further, you need to keep up to date, so that means continually learning: reading, watching, listening.

Second, it’s about sharing. Writing and speaking are the two obvious ways. Sure, you can host a channel: podcast, vlog, blog, but if you’re hosting other folks, you’re seen as well connected but not necessarily as the expert. Further, I reckon you have to be able to write and speak (and pretty much all of these folks do both well). So, start by speaking at small events, and get feedback to improve. Study good presentation style. Then start submitting for events like the Learning Guild, ATD, or LDA (caveats on all of these owing to various relationships, but I think they’re all scrutable). I once wrote about how to read and write proposals, and I think my guidance is still valid.

Similarly, write. Learning Solutions or eLearn Mag are two places to put stuff that’s sensibly rigorous but written for practitioners. Take feedback to heart, and deliberately improve. Make sure you’re presenting value, not pitching anything. What conferences and magazines say about not selling, that your clear approach is what sells, is absolutely true.

Also, make sure that you have a unique ‘voice’. No one needs the same things others are saying, at least in the same way. Have a perspective, your own take. Your brand is not only what you say, but how you say it.

A related comment: track some related fields. Most of the folks I think of as experts have some other area they draw inspiration from. UX/UI, anthropology, software engineering, there are many fields and finding useful insight from a related one is useful to the field and keeps you fresh.

Oh, one other thing. You have to have integrity. People have to be able to trust what you say. If you push something for which you have a private benefit, or something that’s trendy but not real, you will lose whatever careful credibility you’ve built up. Don’t squander it!

So that’s my take on how to be an elearning expert. So, what have I missed?

Overworked IDs

25 May 2021 by Clark 2 Comments

I was asked a somewhat challenging question the other day, and it led me to reflect. As usual, I’m sharing that with you. The question was “How can IDs keep up with everything, feel competent and confident in our work?” It’s not a trivial question! So I’ll share my response to overworked IDs.

There was considerable context behind the question. My interlocutor weighed in with her tasks:

“sometimes I wonder how to best juggle everything that my role requires: project management, design and ux/ui skills, basic coding, dealing with timelines and SMEs and managers. Don’t forget task analysis and needs assessment skills, making content accessible and engaging. And staying on top of a variety of software.”

I recognize that this is the life of overworked IDs, particularly if you’re the lone ID (which isn’t infrequent), or expected to handle course development on your own. Yet it is a lot of different competencies. In work with IBSTPI, where we’re defining competencies, we’re recognizing that different folks cut up roles differently. Regardless, many folks wear different competency requirements that in other orgs are handled by different teams. So what’s a person to do?

My response focused on a couple of things. First, there’re the expectations that have emerged. After 9/11, when we were avoiding travel, there was a push for elearning. And, with the usual push for efficiency, rapid elearning became the vogue. That is, tools that made it easy to take PDFs and PPTs and put it up online with a quiz. It looked like lectures, so it must be learning, right?

One of the responses, then, is to manage expectations. In fact, a recent post addressed the gap between what we know and what orgs should know. We need to reset expectations.

As part of that, we need to create better expectations about what learning is. That was what drove the Serious eLearning Manifesto [elearningmanifesto.org], where we tried to distinguish between typical elearning and serious elearning. Our focus should shift to where our first response isn’t a course!

As to what is needed to feel competent and confident, I’ve been arguing there are three strands. For one (not surprisingly ;), I think IDs need to know learning science. This includes being able to fill in the gaps in and update on instructional design prescriptions, and also to be able to push back against bad recommendations. (Besides the book, this has been the subject of the course I run for HR.com via Allen Academy, will be the focus of my presentation at ATD ICE this summer, and also my asynchronous course for the LDC conference.)

Second, I believe a concomitant element is understanding true engagement. Here I mean going beyond trivial approaches like tarting-up drill-and-kill, and gamification, and getting into making it meaningful. (I’ve run a workshop on that through the LDA, and it will be the topic of my workshop at DevLearn this fall.)

The final element is a performance ecosystem mindset. That is, thinking beyond the course: first to performance support, still on the optimal execution side of the equation. Then we move to informal learning, facilitating learning. Read: continual innovation! This may seem like more competencies to add on, but the goal is to reduce the emphasis (and workload) on courses, and build an organization that continues to learn. I address this in the Revolutionize L&D book, and also my mobile course for Allen Interactions (a mobile mindset is, really, a performance ecosystem mindset!).

If you’re on top of these you should prepared to do your job with competence and confidence. Yes, you still have to navigate organizational expectations, but you’re better equipped to do so. I’ll also suggest you stay tuned for further efforts to make these frameworks accessible.

So, there’re my responses to overworked IDs. Sorry, no magic bullets, I’m afraid (because ‘magic’ isn’t a thing, sad as that may be). Hopefully, however, a basis upon which to build. That’s my take, at any rate, I welcome hearing how you’d respond.

What about books | conferences?

18 May 2021 by Clark Leave a Comment

Responding to a frequent question yet again, I decided to post an answer to the “what about books | conferences?” question.

And, as usual, the transcript.

Once again, after talking about how learning requires meaningful practice, I was asked the seemingly timeless question: “but what about books?” Similarly, I regularly get “what about conferences?” So, for the record, let me say when and why books and lectures make sense. And when not. Hopefully I won’t have to answer another “what about books | conferences?” question.

To start, learning is action and reflection. That is, learning ‘outside’ formal instruction. We act in the world and reflect on it to cement the lesson. It’s slightly more complicated, because certain things, e.g. Geary’s biologically primary things, may not really need reflection. Further some things may be really challenging to learn on your own even with reflection. But basically, doing things and reflecting (which can be reading, experimenting, writing/representing), etc is the way we learn on our own.

Which, as I’ve argued before, suggests that instruction be designed action and guided reflection. That is, instructors should be choosing meaningful activities and scaffolding reflection around it. When we’re designing for novices [link], in particular, when the learner doesn’t know what’s important nor why, we need to do the whole enchilada (darn, now I’m hungry).

Which also means that when we’ve segued beyond novice to practitioner (and beyond), we begin to know what’s important and why, and we just need it. We want resources that can fill in the gaps. We want support for reflection.

So now we can explain why we can attend conferences, read books and articles, and the like. When we’re deeply engaged in something, whether work or a passion, reading a book, listening to someone tell their story, and the like, serves as the necessary adjunct to our activity! They provide the complement to our own endeavors; the reflection to our action!

Now, hopefully, we’ll never again need to discuss this. Realistically, we can point people here when we’ get “what about books | conferences”? At least, that’s my story, what’s yours?

A message to CxOs 2: about org learning myths

11 May 2021 by Clark 2 Comments

When I wrote my last post on a message to CxOs about L&D myths, I got some pushback. Which, for the record, is a good thing; one of us will learn something. As a counter to my claim that L&D often was it’s own worst enemy, there was a counter. The claim was that there are folks in L&D who get it, but fight upward against wrong beliefs. Which absolutely is true as well. So, let’s also talk about what CxOs need to know about the org learning myths they may believe.

First, however, I do want to say that there is evidence that L&D isn’t doing as well as it could and should. This comes from a variety of sources. However, the question is where does the blame lie. My previous post talked about how L&D deludes itself, but there are reasons to also believe in unfair expectations. So here’s the other side.

If it looks like schooling… I used this same one against L&D, but it’s also the case that CxOs may believe this. Further, they could be happy if that’s the case. Which would be a shame just as I pointed out in the other case. Lectures, information dump & knowledge test, in general content presentation doesn’t lead to meaningful change in behavior in the absence of activity. Designed action and guided reflection, which looks a lot more like a lab or studio than a classroom, is what we want.
SMEs know what needs to be learned. Research tells us to the contrary; experts don’t have conscious access to around 70% of what they do (tho’ they do have access to what they know). Just accepting what a SME says and making content around that is likely to lead to a content dump and lack of behavior change. Instead, trust (and ensure) that your designers know more about learning than the SME, and have practices to help ameliorate the problem.
The only thing that matters is keeping costs low. This might seem to be the case, but it reflects a view that org learning is a necessary evil, not an investment. If we’re facing increasing change, as the pundits would have it, we need to adapt. That means reskilling. And effective reskilling isn’t about the cheapest approach, but the most effective for the money. Lots of things done in the name of learning (see above) are a waste of time and money. Look for impact first.
Courses are the answer to performance issues. I was regaled with a tale about how sales folks and execs were insisting that customers wanted training. Without evaluating that claim. I’ll state a different claim: customers want solutions. If it’s persistent skills, yes, training’s the answer. However, a client found that customers were much happier with how-to videos than training for most of the situations. It’s a much more complex story.
Learning stops at the classroom. As is this story. One of the reasons Charles Jennings was touting 70:20:10 was not because of the numbers, but because it was a way to get execs to realize that only the bare beginning came from courses, if at all. There’s ongoing coaching with stretch assignments and feedback, and interacting with other practitioners…don’t assume a course solves a problem. A colleague mentioned how her org realized that it couldn’t create a course without also creating manager training, otherwise they’d undermine the outcomes instead of reinforcing them.
We’ve invested in an LMS, that’s all we need. That’s what the LMS vendors want you to believe ;)! Seriously, if all you’re doing is courses, this could be true, but I’m hoping the above
Customers want training. Back to an earlier statement, customers want solutions. It is cool to go away to training and get smothered in good food and perks. However, it’s also known that sometimes that goes to the manager, not the person who’ll actually be doing the work! Also, training can’t solve certain types of problems. There are many types of problems customers encounter, and they have different types of solutions. Videos may be better for things that occur infrequently, onboard help or job aids may meet other needs to unusual to be able to predict for training, etc. We don’t want to make customers happy, we want to make them successful!
We need ways to categorize people. It’s a natural human thing to categorize, including people. So if someone creates an appealing categorization that promises utility, hey that sounds like a good investment. Except, there are many problems! People aren’t easy to categorize, instruments struggle to be reliable, and vested interests will prey upon the unwary. Anyone can create a categorization scheme, but validating it, and having it be useful, are both surprisingly big hurdles. Asking people questions about their behavior tends to be flawed for complex reasons. Using such tools for important decisions like hiring and tracking have proven to be unethical. Caveat emptor.
Bandwagons are made to be jumped on. Face it, we’re always looking for new and better solutions. When someone links some new research to a better outcome, it’s exciting. There’s a problem, however. We often fall prey to arguments that appear to be new, but really aren’t. For instance, all the ‘neuro’ stuff unpacks to some pretty ordinary predictions we’ve had for yonks. Further, there are real benefits to machine learning and even artificial intelligence. Yet there’s also a lot of smoke to complement the sizzle. Don’t get misled. Do a skeptical analysis. This holds doubly true for technology objects. It’s like a cargo cult, what’s has come down the pike must be a new gift from those magic technologists! Yet, this is really just another bandwagon. Sure, Augmented Reality and Virtual Reality have some real potential. They’re also being way overused. This is predictable, c.f. Powerpoint presentations in Second Life, but ideally is avoided. Instead, find the key affordances – what the technology uniquely provides – and match the capability to the need. Again, be skeptical.

My point here is that there can be misconceptions about learning within L&D, but it can also be outside perspectives that are flawed. So hopefully, I’ve now addressed both. I don’t claim that this is a necessary and complete set, just certain things that are worth noting. These are org learning myths that are worth trying to overcome, or so I think. I welcome your thoughts!

A message to CxOs: about L&D myths

4 May 2021 by Clark 3 Comments

If you’re a CEO, COO, CFO, and the like, are you holding L&D to account? Because much of what I see coming out of L&D doesn’t stand up to scrutiny. As I’ve cited in books and presentations, there’s evidence that L&D isn’t up to scratch. And I think you should know a few things that may be of interest to you. So here’re some L&D myths you might want to watch out for.

If it looks like school, it must be learning. We’ve all been to school, so we know what learning looks like, right? Except, do you remember how effective school actually was? Did it give you many of the skills you apply in your job now? Maybe reading and writing, but beyond that, what did you learn about business, leadership, etc? And how did you learn those things? I’ll bet not by sitting and listening to lectures presented via bulletpoints. If it looks like schooling, it’s probably a waste of time and money. It should look more like lab, or studio.
If we’re keeping our efficiency in line with others, we’re doing good. This is a common belief amongst L&D: well, our [fill in the blank: employees served per L&D staff member | costs per hour of training | courses run per year | etc.] is the same or better than the industry average, so we’re doing good. No, this is all about efficiency, not effectiveness. If they’re not reporting on measurable changes in the improvement of business metrics, like sales, customer service, operations,e tc, they’re not demonstrating their worth. It’s a waste of money.
We produce the courses our customers need. Can they justify that? It’s a frequent symptom that the courses that are asked for have little relation to the actual problem. There are many reasons for performance problems, and a reliable solution is to throw a course at it. Without knowing whether it’s truly a function of lack of skill. Courses can’t address problems like the wrong incentives, or a lack of resources. If you’re not ensuring that you’re only using courses when they make sense, you’re throwing away money.
Job aids aren’t our job. Performance should be the job, not just courses. As Joe Harless famously said: “Inside every fat course there’s a thin job aid crying to get out.” There are many times when a job aid is a better solution than a course. To believe otherwise is one of the classic L&D myths. If they’re avoiding taking that on, they’re avoiding a cheaper and more effective solution.
Informal learning isn’t our job. Well, it might not be if L&D truly doesn’t understand learning, but they should. When you’re doing trouble-shooting, research, design, etc., you don’t know the answer when you start. That’s learning too, and there is a role for active facilitation of best principles. Assuming people know how to do it isn’t justifiable. Informal learning is the key to innovation, and innovation is a necessary differentiation.
Our LMS is all we need. Learning management systems (which is a misnomer, they’re course management systems) manage courses well. However, if they’re trying to also be resource portals, and social media systems, and collaboration tools, they’re unlikely to be good at all that. Yet those are also functions that affect optimal performance and continual innovation (the two things I argue should be the remit of L&D). Further, you want the right tool for the job. One all-singing, all-dancing solution isn’t the way to bet for IT in general, and that holds true for L&D as well.
Our investment in evaluation instruments is valuable. If you’re using some proprietary tools that purport to help you identify and characterize individuals, you’re probably being had. If you’re using it for hiring and promotion, you’re also probably violating ethical guidelines. Whether personality, or behavior, or any other criteria, most of these are methodologically and psychometrically flawed. You’re throwing away money. We have a natural instinct to categorize, but do it on individual performance, not on some flawed instrument.
We have to jump on this latest concept. There’re a slew of myths and misconceptions running around that are appealing and yet flawed. Generations, learning styles, attention spans, neuro-<whatever> and more are all appealing, and also misguided. Don’t spend resources on investing in them without knowing the real tradeoffs and outcomes.These are classic L&D myths.
We have to have this latest technology. Hopefully you’re resistant to new technologies unless you know what they truly will do for your organization. This holds true for L&D as well. They’re as prone to lust after VR and AR and AI as the rest of the organization. They’re also as likely to spend the money without knowing the real costs and consequences. Make sure they’re coming from a place where they know the unique value the technology brings!

There’s more, but that’s enough for now. Please, dig in. Ask the hard questions. Get L&D to be scrutable for real results, not platitudes. Ensure that you’re not succumbing to L&D myths. Your organization needs it, and it’s time to hold them to account as you do the rest of your organization. Thanks, and wishing you all the best.

Something that emerged from a walk, and, well, I had to get it off my chest. I welcome your thoughts.

Evaluating soft skills

27 April 2021 by Clark 3 Comments

As has become a pattern, someone recently asked me how to evaluate soft skills. And without being an expert on soft skill or evaluation, I tried to answer on principle. So I thought about the types of observable data you should expect to find. And that yielded an initial answer. Then I watched an interesting video of a lecture by a scholar and consultant, and it elaborated the challenges. So, there’s a longer answer too. So here’s an extended riff on evaluating soft skills.

I started with wondering what performance outcomes would you expect for soft skills. Coupled, as well, with how could you find evidence of these observable differences. As a short answer, I suggested that there should be 3(+) outcomes from effective soft skills training.

0) the learner should be able to perform in soft skills scenarios (c.f. Will Thalheimer’s LTEM). This is the most obvious. Put them in the situation and ask them to perform. This is the bit that gets re-addressed further down.

1) the learner should be aware of an improvement in their ability to perform. However, asking immediately can lead to a misapprehension of ability. So, as Will Thalheimer advises in his Performance-Focused Smile Sheets, ask them 3 months later. Also, ask about behavior, not knowledge. E.g. “Are you using the <> model in your work, and do you notice an improvement in your ability?”

2) The ‘customers’ of the learner should notice the improvement. Depending on whether that’s internal or external, it might show up (at least in aggregate) in either 360 eval scores, or some observable metric like customer sat scores. It may be harder to collect this data, but of course it’s also more valuable.

3) Finally, their supervisors/managers should notice the improvement, whether observationally or empirically.They should be not only prepared to support the change over time, but asked to look for evidence (including as a basis to fine tune performance).

All together, triangulating on this should be a way to establish the validity.

Now, extending this, Guy Wallace tweeted a link to a lecture by Neil Rackham. In it, Neil makes the case that universities need to change to teaching core skills, in particular the 4 C’s: critical thinking, creativity, communication, and collaboration. He also points out how hard it is to evaluate these without a labor-intensive effort of an individual observing performance. This is a point that others have made, that these skills have hard to observe criteria.

There’s some argument about so-called 21C skills, and yet I can agree that these four things would be good. The question is how to assess them reliably. Rackham argues that perhaps AI can help here. Perhaps, but at this point I’d argue for two things. First, help students self-evaluate (which has the benefits of them understanding what’s involved). Second, instrumenting environments (say, for instance, with xAPI) in which these activities are performed. There will be data records that can be matched to behaviors, initially for human evaluation, but perhaps ultimately for machine evaluation.

Of course, this requires assigning meaningful activities that necessarily involve creativity, critical thinking, communication, and/or collaboration. This means project based work, and I’ve long argued that you can’t learn such skills without a domain. Actually, to create transferable versions, you’d need to develop the skills across domains.

When I teach, I prefer to give group work projects that do require these skills. It was, indeed, hard to mark these extra skills, but I found that scaffolding it (e.g. a ‘how to collaborate’ document) facilitated good outcomes. Being explicit about the best thinking practices isn’t only a good idea, it’s a demonstrably useful approach in general.

So I think developing skills is important. That means we need a means to be evaluating soft skills. We know it when we see it, but it’s hard to necessarily find the opportunity, but if we can assign it, we can evaluate and develop these skills more readily. That, I think, is a desirable goal. What think you?