Learnlets

Secondary

Clark Quinn’s Learnings about Learning

Representation Matters

by 1 Comment

There is a deep sense of where and how representation matters. Then there are less critical, but still important ways in which presentation counts. It includes talking about stereotypes, and calling out inappropriate labeling. Concepts matter, clarity matters, transparency matters. So here are two situations that are worth critiquing.

The first one that struck me this morning was an announcement. A researcher has created a petition asking Pew Research to stop using the ‘generations’ label. They’ve been using it in their research, and yet (as the petition points out) their own research shows it’s problematic.

Now this is a myth I called out in my last book  (specifically on the topic of problematic beliefs). There are several complaints, such as that the boundaries are arbitrary, and the stereotyping is harmful. While we can differ by age, discrepancies are better explained by experience than by ‘generation’.

Another problem came in an article I was connected to on LinkedIn. In it, they were making the case for micro learning. While there are great reasons to tout the benefits of small bits of timely content, they didn’t really distinguish the uses. Which is a problem, since the different uses require different designs.

Here’s where representation matters. Pew Research’s reputation, in my mind, has gone down. I used to fill out some surveys from them, and stopped because the assumptions in the categories they were using were problematic. Finding out that they’re a major proponent of generations only aggravates that. Can I really trust any results they cite when the foundations are flawed?

Similarly, the organization that’s touting micro learning solutions has just undermined any belief in their credibility to actually do this appropriately. When you tout stuff in ways that show you don’t understand the necessary principles, you damage your reputation. I’m not likely to want to use this firm to design my  solutions.

I push strongly for accuracy. This includes evidence-informed design, conceptual clarity, and transparency of motives. If you tout something, do so in a scrutable way. Marketing malarkey only muddies the water, and our industry has enough of a credibility problem.

Yes, there are more important ways representation matters: for kids to see themselves in culturally desirable roles, for voices to be heard. This is a less important aspect, but quality matters. Look at what you are saying, and ensure that it’s worth your audience’s time!

Misaligned expectations

by 1 Comment

As part of the Learning Development Conference that’s going on for the next five weeks (not too late to join in!), there have already been events. Given that the focus is on evidence-based approaches, a group set up a separate discussion room for learning science. Interestingly, though perhaps not surprisingly, our discussion ended up including barriers. One of the barriers, as has appeared in several guises across recent conversations, are the expectations on L&D. Some of them are our own, and some are others, but they all hamper our ability to do our best. So I thought I’d discuss some of these misaligned expectations.

One of the most prominent expectations is around the timeframes for L&D work. My take is that after 9/11, a lot of folks didn’t want to travel, so all training went online. Unfortunately (as with the lingering pandemic), there was little focus on rethinking, and instead a mad rush to get things online. Which meant that a lot of content-based training ended up being content-based elearning. The rush to take content and put it onscreen drove some of the excitement around ‘rapid elearning’.

The continuing focus on efficiency – taking content, adding a quiz, and putting it online – was pushed to the extreme.  It’s now an expectation that with an authoring tool and content, a designer can put up a course in 1-2 weeks. Which might satisfy some box-checking, but it isn’t going to lead to any change in meaningful outcomes. Really, we need slow learning! Yet there’s another barrier here.

Too often, we have our own expectation that “if we build it, it is good”. That is, too often we take an order for a course, we build it, and we assume all is well. There’s no measurement to see if the problem is fixed, let alone tuning to ensure it is. We don’t have expectations that we need to be measuring our impact! Sure it’s hard; we have to talk to the business owners about measurement, and get data. Yet, like other areas of the organization, we should be looking for our initiatives to lead to measurable change. One of these days, someone’s going to ask us to justify our expenditures in terms of impact, and we’ll struggle if we haven’t changed.

Of course, another of our misaligned expectations is that our learning design approaches are effective. We still see, too often, courses that are content-dump, not serious solutions. This is, of course, why we’re talking about learning science, but while one of us had support to be evidence-based, others still do not. We face a populace, stakeholders  and our audiences, that have been to school. Therefore, the expectation is that if it looks like school, it must be learning. We have to fight this.

It d0esn’t help that well-designed (and well-produced) elearning is subtly different than just well-produced elearning. We can’t (and, frankly, many vendors get by on this) expect our stakeholders to know the difference, but we must and we must fight for the importance of the difference. While I laud the orgs that have expectations that their learning group is as evidence-based as the rest, and their group can back that up with data, they’re sadly not as prevalent as we need.

There are more, but these are some major expectations that interfere with our ability to do our best. The solution? That’s a good question. I think we need to do a lot more education of our stakeholders (as well as ourselves). We need to (gently, carefully) generate an understanding that learning requires practice and feedback, and extends beyond the event. We don’t need everyone to understand the nuances (just as we don’t need to know the details of sales or operations or…unless we’re improving performance on it), but we do need them to be thinking in terms of reasonable amounts of time to develop effective learning, that this requires data, and that not every problems has a training solution. If we can adjust these misaligned expectations, we just might be able to do our job properly, and help our organizations. Which, really, is what we want to be about anyway.

Update on my events

by Leave a Comment

In January I posted about my upcoming webinars (now past), workshops, etc. As things open up again (yay, vaccines), some upcoming events will be happening live!  And, of course, virtual. In fact, one starts next week! So I thought it time to update you on the things I’ll be doing. Then we’ll get back to my regular posts ;). So here’s an update on my events.

First, starting next week, is the Learning Development Conference, by the Learning Development Accelerator (caveat: I’m on their advisory board). Last year, it was an experiment. They did several things very well: it was focused on evidence-based approaches, it created timings that worked for a broad section of the world’s populace (e.g. live sessions were offered twice, once early once late), and it had asynchronous content as well as synchronous. It also had ways to maintain contact and discussions. As a result, it was a success, leading to the Accelerator and this second event.

It’s for six weeks, and first I’ve got an asynchronous course on learning science (a subset of the bigger one I do as a blended workshop for HR.com/Allen Academy). I’m also doing two live sessions (at different times) on some of the new results from cognitive science. I’m already dobbed in for one debate, and they’ll likely call on me for more. There are also a suite of the top names in evidence-based L&D appearing doing either or both of live or asynchronous content.

Second, at the end of August, I’ll be speaking at ATD’s International Conference and Exposition. This is a live event in Salt Lake City. (My first since the pandemic!) Of course I’m speaking on learning science; the topic of my book with them. There could even be a book-signing event!  If you don’t know ATD’s ICE, it’s huge, both a blessing and curse. Lots of quality content (ok, mostly ;), almost too many people to find your friends, but lots of new friends to make, with broad coverage. Also, a big exposition (maybe smaller this year ;).

Third, I’ll be at the Learning Guild’s DevLearn again this year. This has always been one of the best conferences because the Guild runs good events (caveat: I’m their first Guild Master). They want it to grow, of course, but as yet it’s still be reasonably sized, and with quality content. For one, I’ll be speaking on learning science implications.

I’ll  also be running a pre-conference workshop on Making Learning Meaningful. And this is, I suggest, truly of interest. I’ve been seeing more and more examples of well-designed content that’s still lacking in engagement, and this workshop is all about that. It’s an area I’ve been actively exploring and synthesizing into practical implications. Like in the series I did on the topic here, I cover how to hook initial interest, then maintain it through the experience. Also considered are the implications for the elements of learning, and a process to make it practical.

I recommend all three (or I wouldn’t be inclined to speak at them). So that’s the current update on my events. Hope to see you at one or another!

Exploring Exploration

by Leave a Comment

Compass  Learning, I suggest, is action and reflection. (And instruction should be  designed action and  guided reflection.) What that action typically ends up being is some sort of exploration (aka experimentation). Thus, in my mind, exploration is a critical concept for learning. That makes it worth exploring exploration.

In learning, we must experiment (e.g. act) and observe and reflect on the outcomes. We learn to minimize surprise, but we also act to generate surprise. I stipulate that we do so when the costs of getting it wrong are low. That is, making learning  safe. So providing a safe sandbox for exploration is a support for learning. Similarly, have low consequences for mistakes generated through informal learning.

However, our explorations aren’t necessarily efficient nor effective. Empirically, we can make ineffective choices such as changing more than one variable at a time, or missing an area of exploration completely. For instruction, then, we need support. Many years ago, Wallace Feurzig argued for  guided exploration, as opposed to free search (the straw man used to discount constructivist approaches). So putting constraints on the task and/or the environment can support making exploration more effective.

Exploration also drives informal learning. Diversity on a team, properly managed, increases the likelihood of searching a broader space of solutions than otherwise. There are practices that increase the effectiveness of the search. Similarly, exploration should be focused on answering questions. We also want serendipity, but there should be guidelines that keep the consequences under control.

By making exploration safe and appropriately constrained, we can advance our understanding most rapidly, either helping some folks learn what others know, or advance what we all know. Exploration is a key to learning, and we need to understand it. Thus, we should also keep exploring exploration!

New recommended readings

by Leave a Comment

My Near Book ShelfOf late, I‘ve been reading quite a lot, and I‘m finding some very interesting books. Not all have immediate take homes, but I want to introduce a few to you with some notes. Not all will be relevant, but all are interesting and even important. I‘ll also update my list of recommended readings. So here are my new recommended readings. (With Amazon Associates links: support your friendly neighborhood consultants.)

First, of course, I have to point out my own Learning Science for Instructional Designers. A self-serving pitch confounded with an overload of self-importance? Let me explain. I am perhaps overly confident that it does what it says, but others have said nice things. I really did design it to be the absolute minimum reading that you need to have a scrutable foundation for your choices. Whether it succeeds is an open question, so check out some of what others are saying. As to self-serving, unless you write an absolute mass best-seller, the money you make off books is trivial. In my experience, you make more money giving it away to potential clients as a better business card than you do on sales. The typically few hundred dollars I get a year for each book aren‘t going to solve my financial woes! Instead, it‘s just part of my campaign to improve our practices.

So, the first book I want to recommend is Annie Murphy Paul‘s The Extended Mind. She writes about new facets of cognition that open up a whole area for our understanding. Written by a journalist, it is compelling reading. Backed in science, it’s valuable as well. In the areas I know and have talked about, e.g. emergent and distributed cognition, she gets it right, which leads me to believe the rest is similarly spot on. (Also her previous track record; I mind-mapped her talk on learning myths at a Learning Solutions conference). Well-illustrated with examples and research, she covers embodied cognition, situated cognition, and socially distributed cognition, all important. Moreover, there‘re solid implications for the redesign of instruction. I‘ll be writing a full review later, but here‘s an initial recommendation on an important and interesting read.  

I‘ll also alert you to Tania Luna‘s and LeeAnn Renninger‘s Surprise. This is an interesting and fun book that instead of focusing on learning effectiveness, looks at the engagement side. As their subtitle suggests, it‘s about how to Embrace the Unpredictable and Engineer the Unexpected. While the first bit of that is useful personally, it‘s the latter that provides lots of guidance about how to take our learning from events to experiences. Using solid research on what makes experiences memorable (hint: surprise!) and illustrative anecdotes, they point out systematic steps that can be used to improve outcomes. It‘s going to affect my Make It Meaningful  work!

Then, without too many direct implications, but intrinsically interesting is Lisa Feldman Barrett‘s How Emotions Are Made. Recommended to me, this book is more for the cog sci groupie, but it does a couple of interesting things. First, it creates a more detailed yet still accessible explanation of the implications of Karl Friston‘s Free Energy Theory. Barrett talks about how those predictions are working constantly and at many levels in a way that provides some insights. Second, she then uses that framework to debunk the existing models of emotions. The experiments with people recognizing facial expressions of emotion get explained in a way that makes clear that emotions are not the fundamental elements we think they are. Instead, emotions social constructs! Which undermines, BTW, all the facial recognition of emotion work.

I also was pointed to Tim Harford‘s The Data Detective, and I do think it‘s a well done work about how to interpret statistical claims. It didn‘t grip me quite as viscerally as the afore-mentioned books, but I think that‘s because I (over-)trust my background in data and statistics. It is a really well done read about some simple but useful rules for how to be a more careful reviewer of statistical claims. While focused on parsing the broader picture of societal claims (and social media hype), it is relevant to evaluating learning science as well.  

I hope you find my new recommended readings of interest and value. Now, what are you recommending to me? (He says, with great trepidation. ;)

The case for model answers (and a rubric)

by 4 Comments

Human body modelAs I‘ve been developing online workshops, I‘ve been thinking more about the type of assessment I want. Previously, I made the case for gated submissions. Now I find another type of interaction I‘d like to have. So here‘s the case for model answers (and a rubric).

As context, many moons ago we developed a course on speaking to the media. This was based upon the excellent work of the principals of Media Skills, and was a case study in my  Engaging Learning book. They had been running a face to face course, and rather than write a book, they wondered if something else could be done. I was part of a new media consortium, and was partnered with an experienced CD ROM developer to create an asynchronous elearning course.  

Their workshop culminated in a live interview with a journalist. We couldn‘t do that, but we wanted to prepare people to succeed at that as an optional extra next step. Given that this is something people really fear (apocryphally more than death), we needed a good approximation. Along with a steady series of exercises going from recognizing a good media quote, and compiling one, we wanted learners to have to respond live. How could we do this?

Fortunately, our tech guy came up with the idea of a programmable answering machine. Through a series of menus, you would drill down to someone asking you a question, and then record an answer. We had two levels: one where you knew the questions in advance, and the final test was one where you‘d have a story and details, but you had to respond to unanticipated questions.  

This was good practice, but how to provide feedback? Ultimately, we allowed learners to record their answers, then listen to their answers and a model answer. What I‘d add now would be a rubric to compare your answer to the model answer, to support self-evaluation. (And, of course, we’d now do it digitally in the environment, not needing the machine.)

So that‘s what I‘m looking for again. I don‘t need verbal answers, but I do want free-form responses, not multiple-choice. I want learners to be able to self-generate their own thoughts. That‘s hard to auto-evaluate. Yes, we could do whatever the modern equivalent to Latent Semantic Analysis is, and train up a system to analyze and respond to their remarks. However, a) I‘m doing this on my own, and b) we underestimate, and underuse, the power of learners to self-evaluate.  

Thus, I‘m positing a two stage experience. First, there‘s a question that learners respond to. Ideally, paragraph size, though their response is likely to be longer than the model one; I tend to write densely (because I am). Then, they see their answer, a model answer, and a self-evaluation rubric.  

I‘ll suggest that there‘s a particular benefit to learners‘ self-evaluating. In the process (particularly with specific support in terms of a mnemonic or graphic model), learners can internalize the framework to guide their performance. Further, they can internalize using the framework and monitoring their application to become self-improving learners.

This is on top of providing the ability to respond in richer ways that picking an option out of those provided. It requires a freeform response, closer to what likely will be required after the learning experience. That‘s similar to what I‘m looking for from the gated response, but the latter expects peers and/or instructors to weigh in with feedback, where as here the learner is responsible for evaluating. That‘s a more complex task, but also very worthwhile if carefully scaffolded.  

Of course, it‘d also be ideal if an instructor is monitoring the response to look for any patterns, but that‘s outside the learners‘ response. So that‘s the case for model answers. So, what say you? And is that supported anywhere or in any way you know?

Overworked IDs

by 2 Comments

I was asked a somewhat challenging question the other day, and it led me to reflect. As usual, I‘m sharing that with you. The question was “How can IDs keep up with everything, feel competent and confident in our work” It‘s not a trivial question! So I‘ll share my response to overworked IDs.

There was considerable context behind the question. My interlocutor weighed in with her tasks:  

“sometimes I wonder how to best juggle everything that my role requires: project management, design and ux/ui skills, basic coding, dealing with timelines and SMEs and managers. Don‘t forget task analysis and needs assessment skills, making content accessible and engaging. And staying on top of a variety of software.”  

I recognize that this is the life of overworked IDs, particularly if you‘re the lone ID (which isn‘t infrequent), or expected to handle course development on your own. Yet it is a lot of different competencies. In work with IBSTPI, where we‘re defining competencies, we‘re recognizing that different folks cut up roles differently. Regardless, many folks wear different competency requirements that in other orgs are handled by different teams. So what‘s a person to do?

My response focused on a couple of things. First, there‘re the expectations that have emerged. After 9/11, when we were avoiding travel, there was a push for elearning. And, with the usual push for efficiency, rapid elearning became the vogue. That is, tools that made it easy to take PDFs and PPTs and put it up online with a quiz. It looked like lectures, so it must be learning, right?

One of the responses, then, is to manage expectations. In fact, a recent post addressed the gap between what we know and what orgs should know. We need to reset expectations.

As part of that, we need to create better expectations about what learning is. That was what drove the Serious eLearning Manifesto [elearningmanifesto.org], where we tried to distinguish between typical elearning and serious elearning. Our focus should shift to where our first response isn‘t a course!  

As to what is needed to feel competent and confident, I‘ve been arguing there are three strands. For one (not surprisingly ;), I think IDs need to know learning science. This includes being able to fill in the gaps in and update on instructional design prescriptions, and also to be able to push back against bad recommendations. (Besides the book, this has been the subject of the course I run for HR.com via Allen Academy, will be the focus of my presentation at ATD ICE this summer, and also my asynchronous course for the LDC conference.)  

Second, I believe a concomitant element is understanding true engagement. Here I mean going beyond trivial approaches like tarting-up drill-and-kill, and gamification, and getting into making it meaningful. (I‘ve run a workshop on that through the LDA, and it will be the topic of my workshop at DevLearn this fall.)

The final element is a performance ecosystem mindset. That is, thinking beyond the course: first to performance support, still on the optimal execution side of the equation. Then we move to informal learning, facilitating learning. Read: continual innovation! This may seem like more competencies to add on, but the goal is to reduce the emphasis (and workload) on courses, and build an organization that continues to learn. I address this in the  Revolutionize L&D book, and also my mobile course for Allen Interactions (a mobile mindset is, really, a performance ecosystem mindset!).

If you‘re on top of these you should prepared to do your job with competence and confidence. Yes, you still have to navigate organizational expectations, but you‘re better equipped to do so. I‘ll also suggest you stay tuned for further efforts to make these frameworks accessible.  

So, there‘re my responses to overworked IDs. Sorry, no magic bullets, I‘m afraid (because ‘magic‘ isn‘t a thing, sad as that may be). Hopefully, however, a basis upon which to build. That‘s my take, at any rate, I welcome hearing how you‘d respond.

What about books | conferences?

by Leave a Comment

Responding to a frequent question  yet again, I decided to post an answer to the “what about books | conferences?” question.

And, as usual, the transcript.

Once again, after talking about how learning requires meaningful practice, I was asked the seemingly timeless question: “but what about books” Similarly, I regularly get “what about conferences”   So, for the record, let me say when and why books and lectures make sense. And when not. Hopefully I won‘t have to answer another “what about books | conferences” question.

To start, learning is action and reflection. That is, learning ‘outside‘ formal instruction. We act in the world and reflect on it to cement the lesson. It‘s slightly more complicated, because certain things, e.g. Geary‘s biologically primary things, may not really need reflection. Further some things may be really challenging to learn on your own even with reflection. But basically, doing things and reflecting (which can be reading, experimenting, writing/representing), etc is the way we learn on our own.  

Which, as I‘ve argued before, suggests that instruction  be designed action and guided reflection. That is, instructors should be choosing meaningful activities and scaffolding reflection around it. When we‘re designing for novices [link], in particular, when the learner doesn‘t know what‘s important nor why, we need to do the whole enchilada (darn, now I‘m hungry).

Which also means that when we‘ve segued beyond novice to practitioner (and beyond), we begin to know what‘s important and why, and we just need it. We want resources that can fill in the gaps. We want support for reflection.

So now we can explain why we can attend conferences, read books and articles, and the like. When we‘re deeply engaged in something, whether work or a passion, reading a book, listening to someone tell their story, and the like, serves as the necessary adjunct to our activity! They provide the complement to our own endeavors; the reflection to our action!

Now, hopefully, we‘ll never again need to discuss this. Realistically, we can point people here when we‘ get “what about books | conferences”? At least, that‘s my story, what‘s yours?  

A message to CxOs 2: about org learning myths

by 2 Comments

When I wrote my last post on a message to CxOs about L&D myths, I got some pushback. Which, for the record, is a good thing; one of us will learn something. As a counter to my claim that L&D often was it’s own worst enemy, there was a counter. The claim was that there are folks in L&D who get it, but fight upward against wrong beliefs. Which absolutely is true as well. So, let‘s also talk about what CxOs need to know about the org learning myths they may believe.  

First, however, I do want to say that there is evidence that L&D isn‘t doing as well as it could and should. This comes from a variety of sources. However, the question is where does the blame lie. My previous post talked about how L&D deludes itself, but there are reasons to also believe in unfair expectations. So here‘s the other side.  

  1. If it looks like schooling… I used this same one against L&D, but it‘s also the case that CxOs may believe this. Further, they could be happy if that‘s the case. Which would be a shame just as I pointed out in the other case. Lectures, information dump & knowledge test, in general content presentation doesn‘t lead to meaningful change in behavior in the absence of activity. Designed action and guided reflection, which looks a lot more like a lab or studio than a classroom, is what we want.
  2. SMEs know what needs to be learned. Research tells us to the contrary; experts don’t have conscious access to around 70% of what they  do (tho’ they do have access to what they know). Just accepting what a SME says and making content around that is likely to lead to a content dump and lack of behavior change. Instead, trust (and ensure) that your designers know more about learning than the SME, and have practices to help ameliorate the problem.
  3. The only thing that matters is keeping costs low.  This might seem to be the case, but it reflects a view that org learning is a necessary evil, not an investment. If we’re facing increasing change, as the pundits would have it, we need to adapt. That means reskilling. And effective reskilling isn’t about the cheapest approach, but the most effective for the money. Lots of things done in the name of learning (see above) are a waste of time and money. Look for impact first.
  4. Courses are the answer to performance issues.  I was regaled with a tale about how sales folks and execs were  insisting that customers wanted training. Without evaluating that claim. I’ll state a different claim: customers want solutions. If it’s persistent skills, yes, training’s the answer. However, a client found that customers were much happier with how-to videos than training for most of the situations. It’s a much more complex story.
  5. Learning stops at the classroom. As is this story. One of the reasons Charles Jennings was touting 70:20:10 was not because of the numbers, but because it was a way to get execs to realize that only the bare beginning came from courses, if at all. There’s ongoing coaching with stretch assignments and feedback, and interacting with other practitioners…don’t assume a course solves a problem. A colleague mentioned how her org realized that it couldn’t create a course without also creating manager training, otherwise they’d undermine the outcomes instead of reinforcing them.
  6. We‘ve invested in an LMS, that‘s all we need. That’s what the LMS vendors want you to believe ;)!  Seriously, if all you’re doing is courses, this could be true, but I’m hoping the above
  7. Customers want training.  Back to an earlier statement, customers want solutions. It is cool to go away to training and get smothered in good food and perks. However, it’s  also known that sometimes that  goes to the manager, not  the person who’ll actually be doing the work! Also, training can’t solve certain types of problems.  There are many types of problems customers encounter, and they have different types of solutions. Videos may be better for things that occur infrequently, onboard help or job aids may meet other needs to unusual to be able to predict for training, etc. We don’t want to make customers happy, we want  to make them successful!
  8. We need ways to categorize people. It’s a natural human thing to categorize, including people. So if someone creates an appealing categorization that promises utility, hey that sounds like a good investment. Except, there are many problems! People aren’t easy to categorize, instruments struggle to be reliable, and vested interests will prey upon the unwary.  Anyone can create a categorization scheme, but validating it, and having it be useful, are both surprisingly big hurdles. Asking people questions about their behavior tends to be flawed for complex reasons. Using such tools for important decisions like hiring and tracking have proven to be unethical. Caveat emptor.
  9. Bandwagons are made to be jumped on. Face it, we’re always looking for new and better solutions. When someone links some new research to a better outcome, it’s exciting. There’s a problem, however. We often fall prey to arguments that appear to be new, but really aren’t. For instance, all the ‘neuro’ stuff unpacks to some pretty ordinary predictions we’ve had for yonks. Further, there are real benefits to machine learning and even artificial intelligence. Yet there’s also a lot of smoke to complement the sizzle. Don’t get misled. Do a skeptical analysis.  This holds doubly true for technology objects. It’s like a cargo cult, what’s has come down the pike must be a new gift from those magic technologists! Yet, this is really just another bandwagon. Sure, Augmented Reality and Virtual Reality have some real potential. They’re also being way overused. This is predictable, c.f. Powerpoint presentations in Second Life, but ideally is avoided. Instead, find the key affordances – what the technology uniquely provides – and match the capability to the need. Again, be skeptical.

My point here is that there can be misconceptions about learning  within  L&D, but it can also be outside perspectives that are flawed. So hopefully, I’ve now addressed both. I don’t claim that this is a necessary and complete set, just certain things that are worth noting. These are org learning myths that are worth trying to overcome, or so I think. I welcome your thoughts!

Evaluating soft skills

by 3 Comments

As has become a pattern, someone recently asked me how to evaluate soft skills. And without being an expert on soft skill or evaluation, I tried to answer on principle. So I thought about the types of observable data you should expect to find. And that yielded an initial answer. Then I watched an interesting video of a lecture by a scholar and consultant, and it elaborated the challenges. So, there‘s a longer answer too. So here‘s an extended riff on evaluating soft skills.

I started with wondering what performance outcomes would you expect for soft skills. Coupled, as well, with how could you find evidence of these observable differences. As a short answer, I suggested that there should be 3(+) outcomes from effective soft skills training.  

0) the learner should be able to perform in soft skills scenarios (c.f. Will Thalheimer’s LTEM). This is the most obvious. Put them in the situation and ask them to perform. This is the bit that gets re-addressed further down.  

1) the learner should be aware of an improvement in their ability to perform. However, asking immediately can lead to a misapprehension of ability. So, as Will Thalheimer advises in his Performance-Focused Smile Sheets, ask them 3 months later. Also, ask about behavior, not knowledge.   E.g. “Are you using the <> model in your work, and do you notice an improvement in your ability”

2) The ‘customers’ of the learner should notice the improvement. Depending on whether that’s internal or external, it might show up (at least in aggregate) in either 360 eval scores, or some observable metric like customer sat scores. It may be harder to collect this data, but of course it‘s also more valuable.  

3) Finally, their supervisors/managers should notice the improvement, whether observationally or empirically.They should be not only prepared to support the change over time, but asked to look for evidence (including as a basis to fine tune performance).  

All together, triangulating on this should be a way to establish the validity.  

Now, extending this, Guy Wallace tweeted a link to a lecture by Neil Rackham. In it, Neil makes the case that universities need to change to teaching core skills, in particular the 4 C‘s: critical thinking, creativity, communication, and collaboration. He also points out how hard it is to evaluate these without a labor-intensive effort of an individual observing performance. This is a point that others have made, that these skills have hard to observe criteria.  

There‘s some argument about so-called 21C skills, and yet I can agree that these four things would be good. The question is how to assess them reliably. Rackham argues that perhaps AI can help here. Perhaps, but at this point I‘d argue for two things. First, help students self-evaluate (which has the benefits of them understanding what‘s involved). Second, instrumenting environments (say, for instance, with xAPI) in which these activities are performed. There will be data records that can be matched to behaviors, initially for human evaluation, but perhaps ultimately for machine evaluation.  

Of course, this requires assigning meaningful activities that necessarily involve creativity, critical thinking, communication, and/or collaboration. This means project based work, and I‘ve long argued that you can‘t learn such skills without a domain. Actually, to create transferable versions, you‘d need to develop the skills across domains.  

When I teach, I prefer to give group work projects that do require these skills. It was, indeed, hard to mark these extra skills, but I found that scaffolding it (e.g. a ‘how to collaborate‘ document) facilitated good outcomes. Being explicit about the best thinking practices isn‘t only a good idea, it‘s a demonstrably useful approach in general.  

So I think developing skills is important. That means we need a means to be evaluating soft skills. We know it when we see it, but it‘s hard to necessarily find the opportunity, but if we can assign it, we can evaluate and develop these skills more readily. That, I think, is a desirable goal. What think you?