strategy Archives - Learnlets

New and improved evaluation

10 April 2018 by Clark 6 Comments

A few years ago, I had a ‘debate’ with Will Thalheimer about the Kirkpatrick model (you can read it here). In short, he didn’t like it, and I did, for different reasons. However, the situation has changed, and it’s worth revisiting the issue of evaluation.

In the debate, I was lauding how Kirkpatrick starts with the biz problem, and works backwards. Will attacked that the model didn’t really evaluate learning. I replied that it’s role wasn’t evaluating the effectiveness of the learning design on the learning outcome, it was assessing the impact of the learning outcome on the organizational outcome.

Fortunately, this discussion is now resolved. Will, to his credit, has released his own model (while unearthing the origins of Kirkpatrick’s work in Katzell’s). His model is more robust, with 8 levels. This isn’t overwhelming, as you can ignore some. Fortunately, there’re indicators as to what’s useful and what’s not!

It’s not perfect. Kirkpatrick (or Katzell? :) can relatively easily be used for other interventions (incentives, job aids, … tho’ you might not tell it from the promotional material). It’s not so obvious how to do so with his new model. However, I reckon it’s more robust for evaluating learning interventions. (Caveat: I may be biased, as I provided feedback.) And should he have numbered them in reverse, which Kirkpatrick admitted might’ve been a better idea?

Evaluation is critical. We do some, but not enough. (Smile sheets, level 1, where we ask learners what they think of the experience, has essentially zero correlation with outcomes.) We need to do a better job of evaluating our impacts (not just our efficiency). This is a more targeted model. I draw it to your attention.

P&D Strategies

4 April 2018 by Clark 1 Comment

In an article, Jared Spool talks about the strategies he sees UX leaders using. He lists 3 examples, and talks about how your strategies need to change depending on where you are in relation to the organization. It made me think about what P&D strategies could and should be.

So, one of the ones he cited isn’t unique to UX, of course. That one is ‘continual mentoring’, always having someone shadowing the top person in a role. He suspects that it might slow things down a bit, but the upside is a continual up-skilling. Back when I led a team, I had everyone have an area of responsibility, but someone backed them up. Cynically, I called it the ‘bus’ strategy, e.g. in case someone was hit by a bus. Of course, the real reason was to account for any variability in the team, to create some collaborative work, to share awareness, and to increase the team’s understanding.. This is an ‘internal’ strategy.

He cites another, about ‘socializing’ the vision. In this one, you are collectively creating the five year vision of what learning (his was UX) looks like. The point is to get a shared buy-in to a vision, but also promotes the visibility of the group. Here again, this is hardly unique to UX, with a small twist ;). This is more an external strategy. I suppose there could be two levels of ‘external’, outside P&D but inside the organization, and then an external one (e.g. with customers).

I’d add that ‘work out loud’ (aka Show Your Work) would be another internal strategy (at least to begin with). Here, the P&D team starts working out loud, with the unit head leading the way. It both gets the P&D team experimenting with the new ways to do things, of course builds shared awareness, and builds a base to start evangelizing outside.

I’d love to hear the strategies you’ve used, seen used, or are contemplating, to continue and expand your ability to contribute to the organization. What’s working? And, for that matter, what’s not?

No all-singing all-dancing solution

3 April 2018 by Clark 1 Comment

I was pinged on LinkedIn by someone who used the entrée of hearing me speak in next week’s Learning Solutions conference to begin discussing LMS capabilities. (Hint: they provide one.) And I thought I’d elaborate on my response, as the discussion prompted some reflections. In short, what are the arguments for and against having a single platform to deliver an ecosystem solution?

In Revolutionize Learning & Development, I argue for a Performance & Development ccosystem. The idea is more than courses, it’s performance support, social, informal, etc. It’s about having a performer-centric support environment that has tools and information to hand to both help you perform in the moment and develop you over time. The goal is to support working alone and together to meet both the anticipated, and unanticipated, needs of the organization.

On principle, I tend to view an ‘all singing all dancing’ solution as likely to fail on some part of that. It’s implausible that a system would have all the capabilities needed. First, there are many functionalities: access to formal learning, supporting access to known or found resources, sharing, collaborating, and more. It’s unlikely that all those can be done well in one platform. Let alone, doing them in ways that matches any one organization’s ways of working.

I’m not saying the LMS is a bad tool for what it does. (Note: I am not in the LMS benchmark business; there are other people that do that and it’s a full time job.) However, can an LMS be a full solution? Even if there is some capability in all the areas, what’s the likelihood that it’s best-of-breed in all? Ok, in some small orgs where you can’t have an IT group capable of integrating the necessary tools, you might settle for working around the limitations. That’s understandable. But it’s different than choosing to trust one system. It’s just having the people act as the glue instead of the system.

It’s always about tradeoffs, and so integrating best-of-breed capabilities around what’s already in place would make more sense to me. For instance, how *could* one system integrate enterprise-wide federated search as a stand-alone platform? It’s about integrating a suite of capabilities to create a performer-centric environment. That’s pretty much beyond a solo platform, intrinsically. Am I missing something?

Return on Wisdom

20 March 2018 by Clark Leave a Comment

In the early days of the internet, I had a chance to read one of the early translations of The Japan That Can Say No. The point I took away was the critique of short-term decision making driven by the need for shareholder returns. I was pondering this the other day, as I wondered how organizations can look to make longer-term investments. And it led me to ponder the question of what the return is on wisdom.

So, the book was a sensation. At the time, the rumor was that it was written by two top Japanese. It wasn’t released in English, but instead was illicitly translated because of the scandalous claims. Still, I thought the assessment of the problems with derivative financing and efficiency approaches were apt. Are these approaches wise?

I use the term wisdom because I think we can, and need to, go beyond ‘smart’. I was pursuing my own quest to transcend what I do, and came upon a nice view of wisdom from Sternberg. This one argues that we should make decisions with both short- and long-term views, for not just us and ours, but for all people, with an explicit consideration of the value that we are following. Ok, so I’m a native Californian, but I don’t see a problem with this view. Smart is ‘in the moment’, wise is looking at the bigger picture.

I ponder this in the context of organizations continually looking to reduce costs through expediency. As an alternative, they could be looking at longer term approaches that help them get their workforce more intrinsically engaged. Does outsourcing and layoffs end up being more costly than investing in better leadership and culture?

There are some answers. Laurie Bassie’s research found that there was a correlation between high scores on handling people and business results. Similarly, Towards Maturity finds that companies with good L&D practices are more likely to be successful. It’s not surprising. When you provide meaningful work with enablement to succeed, you’re aligning the elements to succeed. It’s a path to a coherent organization. And, like with light, it’s more powerful.

There are arguments to move in wise directions. It may be hard if you’re driven by the need to produce short-term returns. Still, it’s the wise thing to do.

Warning: Snake Oil

13 March 2018 by Clark 1 Comment

So this just appeared in my email this morning. Can you tell what’s wrong?

“The next generation of L&D is here. Millennials are quickly becoming the majority of the world’s workforce. They will need training to become successful leaders. But, because they’ve grown up with mobile devices and digital technologies, they learn differently than previous generations. A solid understanding of millennial learning styles will help you create effective training programs.”

Dr Shmoos snake oil

Yep, snake oil. We’ve hit the myth jackpot! How many can you spot? This is the type of stuff marketers come up with that you can’t fight if you don’t know our brains (and more). So let’s take it apart, shall we? (This is so much fun!)

First, ‘next gen of L&D‘. Um, maybe. I don’t see things being done all that differently. It’s more evolution than revolution (despite my exhortations to the latter ;). Still, no real myth yet.

Next, ‘millennials‘. And, yes, we have a winner! The evidence says that there’re no meaningful differences between generations. And if you think about it, it’s much more a continuum than discrete separations. It may be convenient, but it undermines people’s individuality. It’s really a mild form of age discrimination, dealing with people by the year they’re born instead of their unique circumstances. So, we’ve got our first myth. How much would you pay for this? But wait, there’s more!

Need ‘leadership training’? Er, yeah, so does pretty much everyone. Some folks may get there naturally, but that’s not the way to bet. Moving on…

‘Learn differently’ because of growing up digital. This is the ‘digital natives’ story and the ‘digital learning’ story. And both have been debunked. Turns out that folks who’ve grown up with digital technology aren’t necessarily any better at it. They don’t do better searches, for instance. They may be more comfortable, but that’s not what the claim is. Again, this is sort of discrimination, categorizing people by their environment rather than their individual capability.

And, there’s the story that we’re now learning in fundamentally new ways. Er, not. Our brains haven’t evolved that fast. We still need sustained and varied retrieval practice and feedback. No ‘knowledge downloads’ yet. So here we have two, two, two myths in one! (Throwback: who recognizes the reference?) Keeping count? We’re up to three. Now how much would you pay? But wait, there’s still more!

‘Learning styles’. Ow! The zombie that won’t die; kill it, kill it! Back to the evidence: there’s no meaningful and reliable instrument to measure styles, so you couldn’t identify them. And there’s no evidence that adapting to them helps either (which is implicit). So really, this is two more myths! Wow, 5 myths in one paragraph. You’d be hard pressed to do better on purpose!

I worry who might fall prey to this marketing campaign. I hate to tell you this, but there’s no there there. You’d be far better off putting your effort in improving your learning design than buying into this misguided and misleading effort. If you want help with that, let me know (that’s what I do!), and there are plenty of resources (c.f. the Serious eLearning Manifesto).

Folks, my book on myths is coming out at the end of April. You can be prepared to defend yourself for the cost of just a few coffees. And, you can pre-order it now. Our industry needs to get onto a proper basis. This is one small step, but one that needs to be taken.

The Dearth of Science in Learning Technology

21 February 2018 by Clark 4 Comments

Over the years, I’ve looked at a lot of learning technology. And I see a dispiriting trend. There seems to be little learning science of late. What I see are marketing driven decisions, even when there are claims to science! And I think this is a problem.

First, I generally resist the ‘let me show you our product and give us your opinion’. That’s free consulting, and a very rude ask! (Though I’m contemplating it but all they’ll get for free is the number of comments in each category I’ve noted. ;) Still I will investigate things of my volition at times. I end up seeing a lot of technology by checking it out when someone talks about it, or wandering expo halls. And what I see concerns me.

For one, there are too many tools that have suites of features that are oriented towards ‘information dump and knowledge test’. Which we know isn’t going to lead to meaningful learning. Yet when I try to push them to the next level of engagement (cognitive and emotional), they’re uninterested. The response: “this is what our customers say they want”. Which, of course, isn’t what they need.

It gets worse when supposedly more advanced tools are proselytized. I recently sampled one system promoting their advanced memory model. And the free-to-air course on learning science was broken! It failed on a couple of dimensions beyond drilling rote memory about one thing. That’s not a good example to be showing. Yet people who don’t know better might be enthused.

For a quick test, check to see if there’s anyone who understands learning on the executive team of a vendor. You’ll see all the business roles filled. Some might have advisory boards composed of learning folks, but it’s not clear what role they play.

And I get it. Unfortunately, as an industry, we’re not informed consumers. I see continual conceptually fuzzy promotion of ideas, and even societies offering white papers on the latest buzzwords. It’s business, and with business folks in charge (and shareholders to assuage), they’ll do what people want. Yet this isn’t the professionalism we need.

Ok, so this rant doesn’t taint all companies, but it’s too true for many or most. It’s all too easy to look at the typical offerings and point out the fundamental flaws in what they’re doing, if you know how we learn. And you should.

So, I’ll continue on my crusade for us as an industry to lift our game. I hereby offer to assist any learning technology that wants to put it’s money where it’s mouth is to help them understand learning science, build it into their products, and help them promote the benefits. And I likewise offer any organization using learning technology to help them lift their game and be better consumers. I’ve done both before, and am ready to assist others. Because our learners need us to represent their true interests.

Chief Cognitive Officer?

13 February 2018 by Clark Leave a Comment

Businesses are composed of core functions, and they optimize them to succeed. In areas like finance, operations, and information technology, they prioritize investments, and look for continual improvement. But, with the shift in the competitive landscape, there’s a gap that’s being missed. And I’m wondering if a focus on cognitive science needs to be foregrounded.

In the old days, most people were cogs in the machine. They weren’t counted on to be thinking, but instead a few were thinking for the many. And those who could do so were selected on that basis. But that world is gone.

Increasingly, anything that can be automated should be automated. The differentiators for organizations are no longer on the execution of the obvious, but instead the new advantage is the ability to outthink the competition. Innovation is the new watchword. People are becoming the competitive advantage.

However, most organizations aren’t working in alignment with this new reality. Despite mantras like ‘human capital management’ or ‘talent development’, too many practices are in play that are contrary to what’s known about getting the best from people. Outdated views like putting information into the head, squelching discussion, and avoiding mistakes are rife. And the solutions we apply are simplistic.

Ok, so neuroscientist John Medina says our understanding of the brain is ‘childlike’. Regardless, we have considerable empirical evidence and conceptual frameworks that give us excellent advice about things like distributed, situated, and social cognition. We know about our mistakes in reasoning, and approaches to avoid making mistakes. Yet we’re not seeing these in practice!

What I’m suggesting is a new focus. A new area of expertise to complement technology, business nous, financial smarts, and more. That area is cognitive expertise. Here I’m talking about someone with organizational responsibility, and authority, to work on aligning practices and processes with what’s known about how we think, work, and learn. A colleague suggested that L&D might make more sense in operations than in HR, but this goes further. And, I suggest, is the natural culmination of that thought.

So I’m calling for a Chief Cognitive Officer. Someone who’s responsibility ranges from aligning tools (read: UI/UX) with how we work, through designing continual learning experiences, to leveraging collective intelligence to support innovation and informal learning. Doing these effectively are all linked to an understanding of how our brains operate, and having it distributed isn’t working. The other problem is that not having it coordinated means it’s idiosyncratic at best.

One problem is that there’s too little of cognitive awareness anywhere in the organization. Where does it belong? The people closest are (or should be) the L&D (P&D) people. If not, what’s their role going to be? Someone needs to own this.

Digital transformation is needed, but to do so without understanding the other half of the equation is sort of like using AI on top of bad data; you still get bad outcomes. It’s time to do better. It’s a radical reorg, but is it a necessary change? Obviously, I think it is. What do you think?

At the edge

31 January 2018 by Clark 4 Comments

Another response to my request for topics asked about moving from the classroom to the ‘fringe’. Here, I have a very simple response: the case studies in Revolutionize Learning & Development. Each was chosen and structured to talk about the context, specific situation, the plan, the results, and advice. Each also represents a diversity of settings and needs. These represent some folks working at the edge, away from the ‘event’.

Mark Britz, facing more experts than novices, structured his corporate university as a network, not a series of courses. Communities of Practice served as a model for this thinking. This included and Enterprise Social Network and a Knowledge Management system.

Jos Arets and Vivian Heijnin at Tulser talked through a case study working with a medical care organization. The problem was too much hierarchy. Using a Human Performance Improvement approach, they decentralized the work to more self-directed teams. The solution includes continuous assessment, mobile performance support, and coaching.

Coaching also played a role in the case study Jane Bozarth provided. The issue was solving workplace problems. Instead of courses, the solution connected those with demonstrable skills to mentor those who could benefit. A ‘yellow pages’ to find ‘in the moment’ help was also a part.

For an internal self-help solution, Allison Anderson developed a community of practice with events, portal, and a networking platform. Here, the issues was getting disparate groups performing similar functions (L&D) to share best principles.

I had Charles Jennings recount his actions while serving as CLO in a global organization. With a mantra of ‘from event to process’, he used the 70:20:10 framework to rethink a balance of ‘push’ and ‘pull’ services.

In the book, they tell the stories in their own words. They unpack the thinking behind their choices, ‘showing their work’. The contributions are very valuable, and I’m very grateful that they agreed to share them. For that matter, you should find and track these folks!

Each of these were chosen as exemplary of the type of thinking that takes us from the old model to the ‘edge’. We want to be looking holistically at how people think, work, and learn, and aligning our infrastructure (policies, technology, procedures, and culture) accordingly. This is the L&D part of a larger push to make the workplace more effective by making it more humane (read: more aligned with us).

Listening

16 January 2018 by Clark 1 Comment

Listening, as I mentioned, in this case to Guy Wallace. As one of the premier promoters of evidence-based design, he responded to my question about what to post on with:

Any “How Tos” using methods, tools and techniques that you’ve found to work in L&D and Performance Improvement.

Since I am a fan of Guy’s work, I thought I should answer! Now, obviously I don’t work in a typical L&D environment, so this list is somewhat biased. So I mentally ran through memorable projects from the past and looked for the success factors. Besides the best principles I usually advocate, here are a few tips and tricks that I’ve used over the years:

Engage. Obviously, I wrote a book about this, but some of the quick things I do include:
- embed the decisions they should be making in contexts where they make sense
- as Henry Jenkins put it: “put the player in a role they want to be in”
- exaggerate the context
- minimize the distractions
- hook the learners in emotionally from the start
Decisions. I find that working with the objectives for learning projects, it’s critical to focus on the decisions that learners will ultimately be making. I argue that what will make the difference for organizations, going forward, will be better decisions. And it keeps the discussion from focusing on knowledge. Knowledge is needed, but it’s not central.
Brainstorming. When working a strategy session with clients, I seed the discussion before hand with the challenges and background material, and ask that everyone think on their own before we begin collaboration.
Better ‘Pair and Share’. If, in brainstorming, you should think individually before collectively, so should you do so in all forms. So I trialed a ‘pair and share’ where I asked everyone to:
- think on the questions (asking for 2 things) first,
- then share with another,
- and try to reach agreement
- (I polled the first audience I trialed it on, and they said that the discussion was better, FWIW).
Shared language. I have found it valuable, when starting a new project, to run a little ‘presentation’ where I present some of the models that I’m bringing to the table (that’s why I‘m there ;), so we’re starting from a shared understanding. And of course I’ve reviewed materials of theirs beforehand so I can use their terminology. Educating clients is part of a Quinnovation engagement!
Test. In making the Workplace of the Future project with Learnnovators, we were barreling along full tilt, working on the second module, and I was getting increasingly worried about the fact that we hadn’t tested the first. We finally did, relatively informally, but still got valuable feedback that changed our design somewhat. Similarly on other projects, get feedback early and often.
Visualize. My diagramming bent had me map out the workflow of a client’s production process, to identify opportunities to tweak the process to bring in better learning science with minimal interruption. In general, I will often jump up to the whiteboard and try to represent what I’m hearing to see if it’s shared.
Prototype. Similar to the above, I will often mock up what I’m thinking about (in sort of a ‘ape with a crayon’ level of fidelity), to help communicate the idea; e.g. some sort of walkthrough. I find that only a percentage of the audience can imagine what the experience will be without getting somewhat concrete. (And, yes, they do then complain about the production values, despite the tradeoff of cost versus value. Sigh.)
Get the context. I generally try to understand the whole ecosystem (ala ‘the revolution‘) before I engage in specifics. What are the goals, stakeholders, what’s already being done and by whom, etc. It’s important to re-contextualize ‘best principles’, and that requires knowing the context.
Architecture. Thinking through things using a design thinking approach and a systems-thinking perspective, I’ve tried to think of platforms, not just solutions. It might be content architectures, ecosystem elements, but it’s thinking in terms of systems, not just tactics.
Pragmatism. One final approach that has been beneficial is thinking about how to approximate the best with a budget. I used to talk about ‘what would you do if you had magic’, and then see how close you can get with the resources to hand. It’s a heuristic that often has led to an innovative yet viable solution.

Looking at them, I see that they generally reflect my overall focus on aligning what we do with how we think, work, and learn. Your thoughts?

2018 Trajectories

3 January 2018 by Clark Leave a Comment

Given my reflections on the past year, it’s worth thinking about the implications. What trajectories can we expect if the trends are extended? These are not predictions (as has been said, “never predict anything, particularly the future”). Instead, these are musings, and perhaps wishes for what could (even should) occur.

I mentioned an interest in AR and VR. I think these are definitely on the upswing. VR may be on a rebound from some early hype (certainly ‘virtual worlds’), but AR is still in the offing. And the tools are becoming more usable and affordable, which typically presages uptake.

I think the excitement about AI will continue, but I reckon we’re already seeing a bit of a backlash. I think that’s fair enough. And I’m seeing more talk about Intelligence Augmentation, and I think that’s a perspective we continue to need. Informed, of course, by a true understanding of how we think, work, and learn. We need to design to work with us. Effectively.

Fortunately, I think there are signs we might see more rationality in L&D overall. Certainly we’re seeing lots of people talking about the need for improvement. I see more interest in evaluation, which is also a good step. In fact, I believe it’s a good first step!

I hope it goes further, of course. The cognitive perspective suggests everything from training & performance support, through facilitating communication and collaboration, to culture. There are many facets that can be fine-tuned to optimize outcomes.Similarly, I hope to see a continuing improvement in learning engineering. That’s part of the reason for the Manifesto and the Quinnov 8. How it emerges, however, is less important than that it does. Our learners, and our organizations, deserve nothing less.

Thus, the integration of cognitive science into the design of performance and innovation solutions will continue to be my theme. When you’re ready to take steps in this direction, I’m happy to help. Let me know; that’s what I do!