NOTE: the objectives he is referring to is the two main points of his article, clarify the meaning of evaluation, and suggested techniques for conducting the four-level evaluation.

A couple of those don’t look like courses to me…

It is not a tool problem – it is a people problem!

To answer your question about smile sheets being the main source of evaluations – knowing is not doing. For example, starting around 1990, every major magazine of our craft, such as Training Magazine and ASTD, printed articles almost every month hammering the fact that we need to step up our evaluation efforts by going to level 3 and 4. In addition, Jack Phillips was frequently mentioned or wrote articles about the need to do impacts and ROI. Handbooks, such as those produced by ASTD, also explained in detail about the need to raise our levels of evaluation.

But again, knowing is not doing.

As far as changing it, I don’t believe sweeping the four levels under the carpet is the answer because they are still drilled into everyone minds even if do not mention them. I have been through several change initiatives and one thing I know is that you confront and challenge obstacles rather than ignore them because they still persist in one form or another.

Kirkpatrick made two mistakes with his levels – he presented them backwards (which imprinted them upside down in people’s head) and he used reaction rather than what we now know is more important – motivation. Thus, rather than trying to hide it, we transform the tool into a more useful heuristic:

1. What impact (outcome or result) will improve our business?
2. What do the employees have to perform in order to create the desired impact?
3. What knowledge and skills do they need to learn in order to perform? (courses are the LAST answer)
4. What do they need to perceive in order to learn and perform? (do they see a need for the desired performance?)

Yes, the four levels were originally designed for training, but it has evolved over the years. Lets give our profession some credit that if we use the four levels in the proper context, they will be smart enough to realize that we are not offering training as the only solution, but a broad range of performance solutions – and depending on the type of audience, we might have to give them a few hints :-)

And, David, I think if we *do* the abstraction, there’s nothing inherently wrong with the model. ROI can mislead, but that’s a separate issue, I think. If we’re truly focusing level 4 on business goals, we have the argument for the execs. Analytics and data are part of this, for sure, but for once I’ll be the conservative one and say not to throw out the proverbial baby!

1. I often here “training departments don’t measure to level 4” along with excuses of not enough resources, not enough perceived value, etc. I think that shows the misinterpretation and misapplication of the intent.

2. Dan Pontefract said it best in his CLO article: the model has flaws, and hasn’t evolved. I think what Dan was proposing was very powerful- we’ve drawn from this model for too long- we need an evolved or new model that builds upon the core intent.

Don started the conversation and gave us a beginning model. As an industry, we need to continue the discussion instead of just beating up on an imperfect model. And, let’s be honest- businesses ALL run on imperfect models (sales forecasts, financial models…) so, I don’t expect our industry’s model to be perfect either.

So, Kirkpatrick’s model has issues. ROI is a false numbers game. Rainbows and Unicorn…that being said, we often need to clearly show the C-suite how our role in the organization contributes to it’s success.

So, where’s the great new model?

I am excited about this era of big data and analytics. We now have the tools to look at work activity streams, ecosystems, collaborative actions in workspaces and run analytics to find trends. What activities are correlating to success? What discussions are HiPos engaged in? What shared resources are used most by top performers? What communities are successful people in the organization participating in?…

It will take some time to even ask the right questions. Much more (and a lot of guesswork and mistakes) to identify the trends properly. But, we’ve entered an era when we can take measurements of the work and it’s support activities (the embedded learning)- direct from the workstream. And hopefully learn how to participate in that more seamlessly.

I am ready to evolve.

I’d side with the pre-epiphany you in that I’d feel comfortable citing Kirkpatrick whatever the solution in mind. However, just because Kirkpatrick may be wrong in his final analysis, it doesn’t necessarily mean that it undermines the general case, or make his model any less useful.

What about lack of resource as a barrier to more than L1 analysis? It’s my experience that most people are aware of the need to go deeper in their research, but they find it a challenge to do so under pressure to solve the next problem.

I agree that they should modernise the model to incorporate performance support and non-training solutions, but nevertheless I don’t see it as a show stopper. For me, Kirkpatrick is another model that I assume into my own context.

I really like your interpretation of the four levels, Clark, and I doubt whether any learning professional could argue with their importance – whatever they’re labelled!

If Steve Jobs thought the same way – the cell phone is only good for making calls! People need to buy our iPod Touch if they want Apps and buy someone else’s cell phone if they want to make calls. Apple will NEVER make an iPhone! People want to carry two devices and three if they also want a camera!

I see Kirkpatrick’s levels useful for many performance and learning solutions:

1. Is it having the desired impact (outcome or result) that will improve the performance of our business?
2. What do the employees have to performance in order to create the impact?
3. What knowledge and skills do they need to learn in order to perform?
4. What do they need to perceive in order to learn (do they see a need for the desired performance?)?

And none above means they need training (and training does not always mean courses). For example a simple job aid might suffice for supplementing their skills.

Just because something was designed for one thing does not mean it cannot be adapted to do other useful things.