I was defending Kirkpatrick’s levels the other day, and after being excoriated by my ITA colleagues, I realized there was not only a discrepancy between principle and practice, but between my interpretation and as it’s espoused. Perhaps I’ve been too generous.
The general idea is that there are several levels at which you can evaluate interventions:
- whether the recipient considered the intervention appropriate or not
- whether the recipient can demonstrate new ability after the intervention
- whether the intervention is being applied in the workplace, and
- whether the intervention is impacting desired outcomes.
That this is my interpretation became abundantly clear. But let’s start with what’s wrong in practice.
In practice, first, folks seem to think that just doing level 1 (‘smile sheets’) is enough. Far fewer people take the next logical step and assess level 2. When they do, it’s too often a knowledge test. Both of these fail to understand the intention: Kirkpatrick (rightly) said you have to start at level 4. You have to care about a business outcome you’re trying to achieve, and then work backwards: what performance change in the workplace would lead to the desired outcome. Then, you can design a program to equip people to perform appropriately and determine whether they can, and finally see if they like it. And, frankly, level 1 is useless until you finally have had the desired impact, and then care to ensure a desirable user experience. As a standalone metric, it ranks right up there with measuring learning effectiveness by the pound of learners served.
Now, one of the things my colleagues pointed out to me, beyond the failure in implementation, is that Kirkpatrick assumes that it has to be a course. If it’s just misused, I can’t lay blame, but my colleagues proceeded to quote chapter and verse from the Kirkpatrick site to document that the Kirkpatricks do think courses are the solution. Consequently, any mention of Kirkpatrick only reinforces the notion that courses are the salve to all ills.
Which I agree is a mindset all too prevalent, and so we have to be careful of any support that could lead a regression to the status quo. Courses are fine when you’ve determined that a skill gap is the problem. And then, applying Kirkpatrick starting with Level 4 is appropriate. However, that’s more like 15% of the time, not 100%.
So where did I go wrong? As usual, when I look at models, I abstract to a useful level (my PhD focused on this, and Felice Ohrlich did an interesting study that pointed out how the right level of abstraction is critical). So, I didn’t see it tied to courses, but that it could in principle be used for performance support as well (at least, levels 3 and 4). Also for some social learning interventions.
Moreover, I was hoping that by starting at level 4, you’d look to the outcome you need, and be more likely to look at other solutions as well as courses. But I had neglected to note the pragmatic issue that the Kirkpatrick’s imply courses are the only workplace intervention to move the needles, and that’s not good. So, from now on I’ll have to be careful in my reference to Kirkpatrick.
The model of assessing the change needed and working backward is worthwhile, as is doing so systematically. Consequently, at an appropriate level of abstraction, the model’s useful. However, in it’s current incarnation it carries too much baggage to be recommended without a large amount of qualification.
So I’ll stick to talking about impacting the business, and determining how we might accomplish that, rather than talk about levels, unless I fully qualify it.
Clark.
Again you hit the nail on the head (after all every problem is a nail isn’t it?)
Whatever environment we work in as Learning & Development Professionals we need to concentrate on outcomes. I find it’s so hard to get the business to answer the question ‘What change are you hoping to see in the business as a result of this?” I sometimes rephrase this as “Which needles on your dashboard do you need to move?”
The main reason I often see this resistance is that this involves them being able to detect when a needle moves. For that they need to know how they are doing right now so they can see the movement, but so often I see people who don’t know where they are now, they just want it to ‘get better’.
Bill Wilson once taught a full day workshop on ‘Kirkpatrick Upside-Down’ that’s where I learned to look at it from level 4 first just as you do.
It’s better that way, but harder too.
Clark, I find your colleagues’ reaction as being oddly strange. If Dr. Cooper of Motorola thought along those lines we would not have the cell phone – the phone was only designed for the home! It is NOT meant to be carried with you!
If Steve Jobs thought the same way – the cell phone is only good for making calls! People need to buy our iPod Touch if they want Apps and buy someone else’s cell phone if they want to make calls. Apple will NEVER make an iPhone! People want to carry two devices and three if they also want a camera!
I see Kirkpatrick’s levels useful for many performance and learning solutions:
1. Is it having the desired impact (outcome or result) that will improve the performance of our business?
2. What do the employees have to performance in order to create the impact?
3. What knowledge and skills do they need to learn in order to perform?
4. What do they need to perceive in order to learn (do they see a need for the desired performance?)?
And none above means they need training (and training does not always mean courses). For example a simple job aid might suffice for supplementing their skills.
Just because something was designed for one thing does not mean it cannot be adapted to do other useful things.
As with everything, Kirkpatrick is highly dependent on context. Yes, the model is based on the concept of a “course”, but from their perspective I can see that it makes sense to do so. It’s an easy example for the public to understand.
I agree that they should modernise the model to incorporate performance support and non-training solutions, but nevertheless I don’t see it as a show stopper. For me, Kirkpatrick is another model that I assume into my own context.
I really like your interpretation of the four levels, Clark, and I doubt whether any learning professional could argue with their importance – whatever they’re labelled!
Clark
I’d side with the pre-epiphany you in that I’d feel comfortable citing Kirkpatrick whatever the solution in mind. However, just because Kirkpatrick may be wrong in his final analysis, it doesn’t necessarily mean that it undermines the general case, or make his model any less useful.
What about lack of resource as a barrier to more than L1 analysis? It’s my experience that most people are aware of the need to go deeper in their research, but they find it a challenge to do so under pressure to solve the next problem.
Two thoughts here:
1. I often here “training departments don’t measure to level 4” along with excuses of not enough resources, not enough perceived value, etc. I think that shows the misinterpretation and misapplication of the intent.
2. Dan Pontefract said it best in his CLO article: the model has flaws, and hasn’t evolved. I think what Dan was proposing was very powerful- we’ve drawn from this model for too long- we need an evolved or new model that builds upon the core intent.
Don started the conversation and gave us a beginning model. As an industry, we need to continue the discussion instead of just beating up on an imperfect model. And, let’s be honest- businesses ALL run on imperfect models (sales forecasts, financial models…) so, I don’t expect our industry’s model to be perfect either.
So, Kirkpatrick’s model has issues. ROI is a false numbers game. Rainbows and Unicorn…that being said, we often need to clearly show the C-suite how our role in the organization contributes to it’s success.
So, where’s the great new model?
I am excited about this era of big data and analytics. We now have the tools to look at work activity streams, ecosystems, collaborative actions in workspaces and run analytics to find trends. What activities are correlating to success? What discussions are HiPos engaged in? What shared resources are used most by top performers? What communities are successful people in the organization participating in?…
It will take some time to even ask the right questions. Much more (and a lot of guesswork and mistakes) to identify the trends properly. But, we’ve entered an era when we can take measurements of the work and it’s support activities (the embedded learning)- direct from the workstream. And hopefully learn how to participate in that more seamlessly.
I am ready to evolve.
Donald & Dan, you’re doing the same abstraction I unconsciously did. What Kirkpatrick *talks* about is using the framework for training. I agree (and so said) that it could be used more broadly, but I do not want to make it easy for others to slip into using training as the only tool. As my colleague Harold Jarche says: “Where there is a genuine lack of skills and knowledge, training may be required, but it should only be in cases where the other barriers to performance have been addressed”. I think we’re agreeing furiously.
And, David, I think if we *do* the abstraction, there’s nothing inherently wrong with the model. ROI can mislead, but that’s a separate issue, I think. If we’re truly focusing level 4 on business goals, we have the argument for the execs. Analytics and data are part of this, for sure, but for once I’ll be the conservative one and say not to throw out the proverbial baby!
Clark, I’m not sure if I agree with your statement, “but I do not want to make it easy for others to slip into using training as the only tool.” It almost sounds like you mean our profession is not so bright so we have to dumb things down (spoon feed them).
Yes, the four levels were originally designed for training, but it has evolved over the years. Lets give our profession some credit that if we use the four levels in the proper context, they will be smart enough to realize that we are not offering training as the only solution, but a broad range of performance solutions – and depending on the type of audience, we might have to give them a few hints :-)
Donald, i guess I mean that too often, executives, clients, vendors, and more, have either a vested interest in selling a ‘course’ solution, or an obliviousness to the alternative. You’ve got to admit we’re still seeing courses in many instances when they’re not the right solution. That’s what I’m fighting for; bringing a broader repertoire to the table instead of the ‘course’ hammer. That’s what HPT’s about, too. But if you immediately start talking Kirkpatrick, you’ll start talking about a course to impact the metric, and then you end up with people thinking Level 1, or *maybe* Level 2 is sufficient. The proof is in the environment: how long has Kirkpatrick’s model been around, and yet how often do we see smile sheets as the only evaluation? QED
Hi Clark,
I read your latest post, which is very good as it takes this discussion to the next level. Part of the problem is that we believe a tool implies a course, however it is people who imply a course because of their misunderstandings. For example, a few weeks ago I wrote a post on my blog on the misconception that ISD was created to only build courses. I was taught my basic skills by one of the masters of training and development, the U.S. Army, and the manual on ISD (dated 1983) we were given specifically states that a course should only be used if the task cannot be adequately trained elsewhere, such as OJT or Job Performance Aids (I still have the manual). Yet one of the chief complaints about ISD is that ISD was specifically designed to build courses! And the problem is that as you note in your post is a “lack of degree, old-style instruction, myths, templates, the list goes on.”
It is not a tool problem – it is a people problem!
To answer your question about smile sheets being the main source of evaluations – knowing is not doing. For example, starting around 1990, every major magazine of our craft, such as Training Magazine and ASTD, printed articles almost every month hammering the fact that we need to step up our evaluation efforts by going to level 3 and 4. In addition, Jack Phillips was frequently mentioned or wrote articles about the need to do impacts and ROI. Handbooks, such as those produced by ASTD, also explained in detail about the need to raise our levels of evaluation.
But again, knowing is not doing.
As far as changing it, I don’t believe sweeping the four levels under the carpet is the answer because they are still drilled into everyone minds even if do not mention them. I have been through several change initiatives and one thing I know is that you confront and challenge obstacles rather than ignore them because they still persist in one form or another.
Kirkpatrick made two mistakes with his levels – he presented them backwards (which imprinted them upside down in people’s head) and he used reaction rather than what we now know is more important – motivation. Thus, rather than trying to hide it, we transform the tool into a more useful heuristic:
1. What impact (outcome or result) will improve our business?
2. What do the employees have to perform in order to create the desired impact?
3. What knowledge and skills do they need to learn in order to perform? (courses are the LAST answer)
4. What do they need to perceive in order to learn and perform? (do they see a need for the desired performance?)
Clark, I’m sorry for being a pest, but I have to challenge you on the assumption that the four-levels were only designed for courses. I’m looking at the 1994 edition of “The ASTD Training and Development Handbook,” edited by Robert Craig. On page 294 there is an article by Donald Kirkpatrick and he writes, “These objectives will be related to in-house classroom programs, one of the most common forms of training. Many of the principles and procedures applied to all kinds of training activities, such as performance review, participation in outside programs, programmed instruction, and the reading of books.”
NOTE: the objectives he is referring to is the two main points of his article, clarify the meaning of evaluation, and suggested techniques for conducting the four-level evaluation.
A couple of those don’t look like courses to me…
Donald, appreciate the dialog. And pleased if Don K did originally talk about other applications than training. But if you go to their site: http://www.kirkpatrickpartners.com/ you’ll see training being the dominant discussion, and the implication of business results from training can be inferred. As I was soundly informed by my colleagues ;). Absolutely right that the levels can be applied more appropriately. I might reframe your #3 to “what knowledge, skills, and resources do they need in order to perform?” Thanks for engaging!