Learnlets

A few years ago, I had a ‘debate’ with Will Thalheimer about the Kirkpatrick model (you can read it here).  In short, he didn’t like it, and I did, for different reasons.  However, the situation has changed, and it’s worth revisiting the issue of evaluation.

In the debate, I was lauding how Kirkpatrick  starts with the biz problem, and works backwards. Will attacked that the model didn’t really evaluate learning. I replied that it’s role wasn’t evaluating the effectiveness of the learning design on the learning outcome, it was assessing the impact of the learning outcome on the organizational outcome.

Fortunately, this discussion is now resolved. Will, to his credit, has released his own model (while unearthing the origins of Kirkpatrick’s work in Katzell’s).  His model is more robust, with 8 levels.  This isn’t overwhelming, as you can ignore some. Fortunately, there’re indicators as to what’s useful and what’s not!

It’s not perfect. Kirkpatrick (or Katzell? :) can relatively easily be used for other interventions (incentives, job aids, … tho’ you might not tell it from the promotional material). It’s not so obvious how to do so with his new model.  However, I reckon it’s  more robust for evaluating learning interventions. (Caveat: I may be biased, as I provided feedback.) And  should he have numbered them in reverse, which Kirkpatrick admitted might’ve been a better idea?

Evaluation is critical.  We do some, but not enough. (Smile sheets, level 1, where we ask learners what they think of the experience, has essentially zero correlation with outcomes.) We need to do a better job of evaluating our impacts (not just our efficiency). This is a more targeted model.  I draw it to your attention.

In an article, Jared Spool talks about the strategies he sees UX leaders using.  He lists 3 examples, and talks about how your strategies need to change depending on where you are in relation to the organization.  It made me think about what P&D strategies could and should be.

So, one of the ones he cited isn’t unique to UX, of course. That one is ‘continual mentoring’, always having someone shadowing the top person in a role. He suspects that it might slow things down a bit, but the upside is a continual up-skilling.  Back when I led a team, I had everyone have an area of responsibility, but someone backed them up.  Cynically, I called it the ‘bus’ strategy, e.g. in case someone was hit by a bus.  Of course, the real reason was to account for any variability  in the team, to create some collaborative work, to share awareness,  and to increase the team’s understanding..  This is an ‘internal’ strategy.

He cites another, about ‘socializing’ the vision.  In this one, you are collectively creating the five year vision of what learning (his was UX) looks like. The point is to get a shared buy-in to a vision, but also promotes the visibility of the group.  Here again, this is hardly unique to UX, with a small twist ;).  This is more an external strategy.  I suppose there could be two levels of ‘external’, outside P&D but inside the organization, and then an external one (e.g. with customers).

I’d add that ‘work out loud’ (aka Show Your Work) would be another internal strategy (at least to begin with).  Here, the P&D team starts working out loud, with the unit head leading the way. It both gets the P&D team experimenting with the new ways to do things, of course builds shared awareness,  and builds a base to start evangelizing  outside.

I’d love to hear the strategies you’ve used, seen used, or are contemplating, to continue and expand your ability to contribute to the organization.  What’s working?  And, for that matter, what’s not?