Learnlets

Secondary

Clark Quinn’s Learnings about Learning

Malicious metrics

by Leave a Comment

Like others, I have been seduced by the “what X are you” quizzes on FaceBook. I certainly understand why they’re compelling, but I’ve begun to worry about just why they’re so prevalent.  And I’m a wee bit concerned.

People like to know things about themselves. Years ago, when we built an adaptive learning system (it would profile you versus me, and then even if we took the same course we’d be likely to have a different experience), we realized we’d need to profile learning  a priori.  That is, we’d ask an initial suite of questions, and that’d prime the system. (And we  intended this profiling to be a game, not a set of quiz questions). Ultimately that initial model built by the questions would get refined by learner behavior  in  the system (and we also intended a suite of interventions ‘layered’ on top that would help improve learner characteristics that were malleable).

The underlying mission given us by my CEO was to help learners understand themselves  as learners, and use that to their advantage.  So, in addition to asking the questions, we’d share with them what we’d learned about them as learners.  The notion was what we irreverently termed the ‘Cosmo quiz’, those quizzes that appeared in Cosmopolitan magazine about “how good a Y” you are, where one  takes quizzes and then adds up the score.

Fast forward to now, and I began to wonder about these quizzes. They seem cute and harmless, but without seeing all the possible outcomes, it certainly seemed like it might not take that many questions to determine which one you’d qualify as.  Yes, in good test design, you ask a question a number of times to disambiguate.  But it occurred to me that you could use fewer questions (and the outcomes are always written intriguingly so you don’t necessarily mind which you become), and then wonder what are the other questions being used for.  And the outcomes here don’t really matter!

So, it’d be real easy to insert demographic questions and use that information (presumably en masse) to start profiling markets.  If you know other information about these people, you can start aggregating data and mining for information.  One question  I saw, for instance, asked you to pick which setting (desert, jungle, mountain, city), etc.  Could that help recommend vacations to you?

When I researched these quizzes, rather than finding concerns about the question data, instead I found that much more detailed information about your account  was allowed to be passed from Facebook  to  the quiz hoster.  Which is worse!  Even if not, I begin to worry that while they’re fun, what’s the motivation to keep creating new ones?  What’s the business relationship?  And I think it’s data.

Now, getting better data means you might get more targeted advertising.  And that might be preferable than random (I’ve seen some pretty funny complaints about “what made them think this was for me”).  But I don’t feel like giving them that much insight. So I’m not doing any more of those.  I don’t think they really know what animal/movie character/color/fruit/power tool  I am.  If you want to know, ask me.

From Content to Experience

by 1 Comment

A number of years ago, I said that the problem for publishers was not going from text to content (as the saying goes), but from content to experience.  I think elearning designers have the same problem: they are given a knowledge dump, and have to somehow transform that into an effective experience.  They may even have read the Serious eLearning Manifesto, and want to follow it, but struggle with the transition or transformation.  What’s a designer to do?

The problem is, designers will be told, “we need a course on this”, and given a dump of Powerpoints (PPTs), documents (PDFs), and maybe access to a subject matter expert (SME).  This is all about knowledge.  Even the SME, unless prompted carefully otherwise, will resort to telling you the knowledge they’ve learned, because they just don’t have access to what they know.  And this, by itself, isn’t a foundation for a course.  Processing the knowledge, comprehending it, presenting it, and then testing on acquisition (e.g. what rapid elearning tools make easy), isn’t going to lead to a meaningful outcome. Sorry, knowledge isn’t the same as ability to perform.

And this ignores, of course, whether this course is actually needed.  Has anyone checked to see that if the skills associated with this knowledge have a connection with a real workplace performance issue?  Is the performance need a result of a lack of skills?  And is this content aligned to that skill?  Too often folks will ask  for a course on X when the barrier is something else.  For instance, if the content is a bunch of knowledge that somehow you’re to magically put in someone’s head, such as product information or arbitrary rules, you’re far better off putting that information in the world than trying to put it in the head.  It’s really hard to get arbitrary information in the head.  But let’s assume that there is a core skill and workplace need for the sake of this discussion.

The key is determining what this knowledge actually supports  doing differently.  The designer needs to go through that content and figure out what individuals will be able to  do that they can’t do now (that’s important), and then develop practice doing that. This is so important that, if what they’ll be able to do differently, isn’t there, there should be push back.  While you can talk to the SME (trying to get them to talk in terms of decisions they can make instead of knowledge), you may be better off inferring the decisions and then verifying and refining with the SME.  If you have access to several SMEs, better yet get them in a room together and just facilitate until they come up with the core decisions, but there are many situations where that’s not feasible.

Once you have that key decision, the application of the skill in context, you need to create situations where learners can practice using it.  You need to create scenarios  where these decisions will play out. Even just better written multiple choice questions that have: story setting, situation precipitating decision, decision alternatives that are ways in which learners might go wrong,  consequences of the decisions, and feedback.  These practice attempts are the core of a meaningful learning experience. And there’s even evidence that putting problems up front or at core is a valuable practice.  You also want to have sufficient practice not just ’til they get it right, but until they have a high likelihood of not getting it wrong.

One thing that might not be in the PDFs and PPTs are examples.  It’s helpful to get colorful examples of someone using  information to successfully solve a problem, and also cases where they misapplied it and failed.  Your SME should be able to help you here, telling you engaging stories of wins and losses.  They may be somewhat resistant to the latter; worst case have them tell them about someone else.

The content in the PDFs and PPTs then gets winnowed down into just the resource material that helps the learner actually able to do the task, to successfully make the decision.  Consider having the practice set in a story, and the content is available through the story environment (e.g. casebooks on the shelves for examples, a ‘library’ for concepts).  But even if you present the (minimized) content and then have practice, you’ve shifted from knowledge dump/test to more of a flow of experience.  The suite  of  meaningful practice, contextualized well and made meaningful with a wee bit of exaggeration and careful alignment with learner’s awareness, is the essence of experience.

Yes, there’s a bit more to it than that, but this is the core: focus on  do, not dump.  And, once you get in the habit, it shouldn’t  take longer, it just takes a change in thinking.  And even if it does, the dump approach isn’t liable to  lead to any meaningful learning, so it’s a waste of time anyway.  So, create experiences, not content.

 

Vale Don Kirkpatrick

by 2 Comments

Last week, Don Kirkpatrick passed away.  Known for his four ‘levels‘ of measuring learning, he’s been hailed and excoriated.  And it’s instructive to see why on both sides.

He derived his model as an approach to determine the impact of an intervention on organizational performance.  He felt that you worked backward from the change you needed, to determine whether the workplace performance was changing, as then to see if that could be attributed to the training, and ultimately to the learner.  He numbered his steps so that step 1 was seeing what learners thought, 2 was that learners  could demonstrate a change, 3 was that the change was showing up in the workplace post intervention, and 4 was it impacting business measures.

This actually made a lot of sense. Rather than measuring the cost of hour of seat time or some other measure of efficiency, or, worse, not measuring at all, here was a plan that was designed to focus on meaningful change that the business needed.  It was obvious, and yet also obviously needed.    So his success in bringing awareness to the topic of business impact is to be lauded.

There were two major  problems, however.  For one, having numbered it the way that it was, people seemed that they could take a partial attempt.  Research shows that the number of people would only do step 1 or 2, and these are useless without ultimately including 4.  He even later wondered if he should have numbered the approach in the reverse.  The numbers have been documented (from a presentation with results from the ASTD Benchmarking Forum) as dropping in implementation from 94% doing level 1, 34% doing level 2, 13% doing level 3, and 3% doing level 4.  That’s  not  the idea!

The second problem was that whether or not he intended it (and there are reasons to believe he didn’t), it become associated only with training interventions.  Performance support interventions or social network outcomes  could similarly be measured (at least on levels 3 and 4), yet the language was all about training, which made it easy for folks to wrongly conclude that training was your only tool.  And we still see folks using courses as the only tool in their repertoire, which just isn’t aligned with how we think, work, and learn (hence the revolution).

Kirkpatrick rode this tool for the rest of his career,  created a family business in it, and he wasn’t shy about suggesting that you buy a book to learn about it.  I certainly can’t fault him for it either, as he did have a sensible model and it could be put into effective use.  There are worse ways to earn a living.

Others have played upon his model.  The Phillips have made a similar career with their fifth level, ROI, measuring the cost of impacting level 4 against the value of the impact.  Which isn’t a bad move to make  after you focus on making an impact.  Similarly, a client opined that there was also level 0, are the learners even showing up for the training!

In assessing the impact, part of me is mindful that tools can be used for good or ill.  Powerpoint doesn’t kill people, people do, as the saying goes.  Still, Kirkpatrick could’ve renumbered the steps, or been more outspoken about the problems with just step 1.

So, I laud his insight, and bemoan the ultimate lack of impact.  However, I reckon it’s better to argue about it than be ignorant.  Rest in peace.

‘Sharing’ culture

by Leave a Comment

I was in a recent conversation about a company facing strong growth and worried about the impact on culture.  Companies with a positive culture, a valuable offering, and a good business model are liable to face growth issues, and maintaining or starting a good culture becomes a critical issue to maintaining the organization’s success.

This company had a positive culture, in that people were diverse, friendly, upbeat, and committed to contributing. These are all positive elements that had led to the early success. Growth, both through hiring and acquisitions, was leading to concerns about the ability for those factors to continue.

One of the things that wasn’t obvious from the initial portrayal of the company was whether folks there were capturing and sharing what they were doing, how they were working, what challenges they were facing, and what results they were seeing. In a small company, this happens naturally through conversation, but face to face communication isn’t scalable.

One obvious possibility is to implement or more systematically leverage an enterprise social network (ESN; essentially    using social media in the org).  Working out loud, as it’s known, has many benefits.  As people share their work, others can comment and improve it.  People can ask for help and get collaboration on those new problems and innovation needs that are increasingly arising.  Mistakes can be made and the lessons learned can be shared so no others have to make the same mistakes.

One of the offshoot benefits of such sharing is that it takes the positive cultural attributes already being shown and makes them visible (if implicitly) as well.  It’s not guaranteed, but with an awareness of the behaviors and manifestations of culture through the network, a systematic process could lead to that positive culture scaling  and yield those additional benefits that accompany working out loud.

It takes all the elements of a learning culture and organizational change, of course. You need to continue to welcome diversity, be open to new ideas, and have it safe to contribute.  You also need to develop a vision, message it, have the leadership model it, facilitate it, anticipate problems and be prepared to address them, and ultimately reward the desired outcomes.  But this is doable.

The benefits of a positive culture are becoming known, and the value of social networks are also emerging. Linking them together is not only necessary, but the benefits are more than the sum of the parts.

#itashare

Facilitating Innovation

by 4 Comments

One of the things that emerged at the recent A(S)TD conference was that a particular gap might exist. While there are resources about learning design, performance support design, social networking, and more, there’s less guidance about facilitating innovation.  Which led me to think a wee bit about what might be involved.  Here’s a first take.

So, first, what are the elements of innovation?  Well, whether you  listen to Stephen Berlin Johnson on the story of innovation, or Keith Sawyer on ways to foster innovation, you’ll see that innovation isn’t individual.  In previous work, I looked at models of innovation, and found that either you mutated an existing design, or meld two designs together.  Regardless, it comes from working and playing well together.

The research suggests that you  need to make sure you are addressing the right problem, diverge on possible solutions via diverse teams under good process, create interim representations, test, refine, repeat.  The point being that the right folks need to work together over time.

The barriers are several.  For one, you need to get the cultural elements right: welcoming diversity, openness to new ideas, safe to contribute, and time for reflection.  Without being able to get the complementary inputs, and getting everyone to contribute, the likelihood of the best outcome is diminished.

You also shouldn’t take for granted that everyone knows how to work and play well together.  Someone may not be able to ask for help in effective ways, or perhaps more likely, others may offer input in ways that minimize the likelihood that they’ll be considered.  People may not use the right tools for the job, either not being aware of the full range (I see this all the time), or just have different ways of working. And folks may not know how to conduct brainstorming and problem-solving processes effectively  (I see this as well).

So, the facilitation role has many opportunities to increase the quality of the outcome.  Helping establish culture, first of all, is really important.  A second role would be to understand and promote the match of tools to need. This requires, by the way, staying on top of the available tools.  Being concrete about learning and problem-solving processes, and  educating them and looking for situations that need facilitation, is another role  Both starting up front and educating folks before these skills are needed are good, and then monitoring for opportunities to tune those skills are valuable.  Finally, developing process facilitation skills,  serving in that role or developing the skills, or both, are critical.

Innovation isn’t an event, it’s a process, and it’s something that I want P&D (Learning & Development 2.0 :) to be supporting. The organization needs it, and who better?

#itashare

Beyond Talent Development

by 5 Comments

On Tuesday, with a big presentation, the American Society for Training & Development announced a rebranding. The new name is the Association for Talent Development, going from ASTD to ATD. And while this is a necessary move, I think it wasn’t the best change they could’ve made.

ASTD needed the change, for two reasons. For one, ASTD has membership in, and runs events, around the world. There may be other orgs (e.g. CSTD for Canada), but the 800 lb gorilla is ASTD. Second, training, while still a large proportion of what ASTD does (rightly or wrongly), is increasingly being joined by other approaches such as coaching and mentoring.

The first reason resonates, but I have a problem with the second. To put it another way, I believe that the change to Association makes sense, but Talent Development doesn’t.  As I stated in Revolutionize Learning & Development, I believe that the necessary direction for organizations is to couple optimal performance with continual innovation. What’s required from Learning & Development, then, is to support all manners of performance and develop continual innovation.

What’s involved is not only to support training when it needs to be ‘in the head’, but using performance support when we can. And we need to develop and facilitate organizational innovation. The latter means not only developing individual (and group) ability to interact constructively, but to facilitate useful interactions of all sorts.

And here’s the rub. I see Talent Development as developing people through training, mentoring, and coaching, but I see the potential role for the folks in what now is termed L&D to be not only the development of people’s ability, but their ability to perform, even if it isn’t developing the person. That is, using performance support when it makes sense should be part of the unit’s responsibility, even when it doesn’t develop the person. Similarly, I see facilitating constructive interaction (curating resources, removing barriers to interaction and supporting tool use, etc), whether it develops people or not, as a vital role.

That’s the reason I chose to suggest, in the book, that the unit should be renamed Performance & Development; supporting both optimal execution and continual innovation in all relevant ways. The opportunity is to be the strategic organizational resource to ensure that all the intellectual resources of the organization are contributing.

And that is the reason I have a problem with Talent Development. To me, Talent Development is focused only on developing people instead of facilitating overall organization performance. And I think that’s falling short of the opportunity, and the need. Don’t get me wrong, I laud that ASTD made a change, and I think Talent Development is a good thing. Yet I think that our role can and should be more.  I  wish they’d thought a little broader, and covered all of the potential contributions.  So, maybe, Association of Performance & Development or APD. Regardless, it’s a dynamic organization that offers a lot. I just wonder who’s going to fill the gaps.

Learning Quotient?

by 2 Comments

I had the good fortune to be invited to the Future of Work event that was held here in Silicon Valley two  weeks ago, and there were four breakouts, one of which was on learning and knowledge management.  You can guess which one I was on (though tempted by the leadership and culture one; there was overlap).

Within that breakout the activity was to pick four topics and further break out.  The issue was meeting workplace needs given the changing nature of work, and I suggested that perhaps the biggest need was to focus on skills that held true across domain, so called meta-cognitive skills, and learning to learn (a total surprise, right?).  That was one that people were interested in, so that’s what we discussed.

We broke down learning into some component elements.  We talked about how your beliefs about learning (your epistemological stance) mattered, as well as your intention to learn, and how effective you were at learning alone and with others.  It also matters how well you use tools and external representations, as well as your persistence.

What emerged was that learning skills shouldn’t be taken for granted.  And consequently, one of the attendees suggested that perhaps along with IQ and EQ, we should be looking at people’s LQ or learning quotient.  I just saw an advertisement that said EQ (they called it EI Emotional Intelligence, probably to avoid trademark infringement) was better than IQ because you can improve your EQ scores. However, the evidence suggests you can improve your LQ scores too.

A decade ago now, Jay Cross and I were pushing the meta-learning lab, and I still think Jay was right claiming that meta-learning might be your best investment.  So, are you aware of how you learn? Have you improved how you learn?  Can you help others learn more effectively?  I believe the answer is yes, and we not only can, but  should.

No, we’re not “doing all right”

by 1 Comment

A number of the Change Agents Worldwide  resonated with an image of the 50 reasons not to change at work that was overlaid on an image of Doug Engelbart. The point, of course, is that all these reasons not to change provide a significant barrier to success. We decided that we‘d line up and  start a blog carnival with each of us addressing one. I decided to address number 45:

45. We‘re doing all right as it is.

First, do you have evidence for that, or are you in denial? It‘s easy to not want to change because it‘ll be hard, so it‘s easier to say that things are OK. It is particularly easy if you aren‘t really checking! L&D, for instance, has been notoriously bad about seeing whether their interventions are actually achieving any impact (around 3% reporting that they actually go to level 4 on Kirkpatrick‘s scale).

If you are checking, what are your benchmarks? Are you measuring just execution, or are you including innovation? Because, as I‘ve said before, optimal execution is only going to let you survive, to thrive you‘ll need continual innovation. If you‘re doing what your strategy says, are you looking out for disruptive forces, and creating your own? You should be checking to see if you‘re striking that balance of being able agile enough as well as productive enough.

Finally, is “all right” really good enough? Shouldn‘t we be shooting for the best, not just good enough? Do you think your competitors are sitting complacent? We really should be looking for every edge. That doesn‘t come from believing we‘re doing all right. We should be looking for continual improvement.

Yes, it‘s harder to not believe we‘re doing all right as it is, but your curiosity should be driving you forward regardless. Your organization, if it isn‘t continually learning, is declining. Are you really doing all right? If you‘re definition of ‘all right‘ is that you are continually curious and moving forward with experimentation, then I reckon so.

Revolution hits the road

by Leave a Comment

Finally!    Revolutionize Learning & Development (Performance and Innovation Strategy for the Information Age) is now shipping,  (and has been available on Kindle for a couple of weeks), so at last you should be able to get your mitts on it if you’re so inclined.  And, if you’re at all involved in trying to make your organization successful, I will immodestly suggest you might want to be so inclined.

Just as background, it documents my claim that organizational L&D isn’t doing what it could and should be doing, and what it  is doing, it is doing badly. Then it lays out  elements that should be considered, what it would look like if it were going well, and    how you  might get there.  While the exact strategy for any one organization will be dependent on where they are and the nature of the business, there are some frameworks to help you apply those to your business.  The goal is to move to Performance & Development, coupling optimal execution with continual innovation.

If you’re curious but not yet ready to dig in, let me mention a couple of things:

  • there’s a free sample available
  • I was interviewed by GameOn Learning’s Bryan Austin about the topic (shorter and more focused)
  • I was also interviewed by Rick Zanotti of Relate (longer and more wide-ranging)

And, if you’re going to be at ASTD’s International Conference  in DC next week, I will be there and:

  • presenting about the issues  on Monday the 5th of May  at 4:30 PM in room 146B (session M313)
  • signing copies right after (5:30 PM) in the ASTD book store
  • holding an author chat on Wednesday May 7th at 9:45 AM also at the ASTD Bookstore

So, check it out, and see if it makes sense to you.  Or you can just  go ahead and get it.  I hope to see you in DC, and welcome your feedback!