Search Results for: Iterating and evaluating

I’ve argued before about the need for evaluation in our work. This occurs summatively, where we’re looking beyond smile sheets to actually determine the impact of our efforts. However, it also should work formatively, where we’re seeing if we’re getting closer. Yet there are some ways in which we go off track. So I want to talk about iterating and evaluating our learning initiatives.

Let’s start by talking about our design processes. The 800 lb gorilla of ADDIE has shifted from a water flow model to a more iterative approach. Yet it still brings baggage. Of late, more agile and iterative approaches have emerged, not least Michael Allen’s SAM and Megan Torrance’s LLAMA. Agile approaches, where we’re exploring, make more sense when designing for people, with their inherent complexity.

Agile approaches work on the basis of creating, basically, Minimum Viable Products, and then iterating.  We evaluate each iteration. That is, we check to see what need to be improved, and what is good enough. However,  when are we done?

In my workshops, when talking about iteration, I like to ask the audience this question. Frequently, the answer is “when we run out of time and money”. That’s an understandable answer, but I maintain it’s the  wrong answer.

If we iterate until we run out of time and money, we don’t know that we’ve actually met our goals. As I explained about social media metrics, but applies here too, you  should be iterating until you achieve the metrics you’ve set. That means you know what you’re trying to do!

Which requires, of course, that you set metrics about what your solution should achieve. That could include usability and engagement (which come before and after, respectively), but most critically ‘impact’. Is this learning initiative solving the problem we’re designing it to achieve?  Which also means you need to have a discussion of why you’re building it, and how you know it’s working.

Of course, if you’re running out of time and money faster than you’re getting close to your goal, you have to decide whether to relax your standards, or apply for more resources, or abandon your work, or…but at least you’re doing so consciously. Yet this is still better than heuristically determining that three iterations is arbitrarily appropriate, for example.

I do recognize that this isn’t our current situation, and changing it isn’t easy. We’re still asked to make slide decks look good, or create a course on X, etc. Ultimately, however, our professionalism will ask us to do better. Be ready. Eventually, your CFO should care about the return on your expenditures, and it’ll be nice to have a real answer. So, iterating and evaluating  should  be your long term approach. Right?

We (the Learning Development Accelerator) just released Guy Wallace’s latest tome, The L&D Pivot Point. Then, we had an interview with him to explain what it’s about. Despite having a ring-side seat (I served as editor, caveat emptor), it was eye-opening to hear him talk about what it’s about! It really is about the pivotal point in L&D, when you move from just offering courses to looking at performance. It’s such an important point that it’s worth reiterating.

So, the official blurb for the book talks about his tried and tested processes. In the interview, he talks about how he’s synthesized the work of the leaders of the performance improvement movement, people like Joe Harless, Geary Rummler, Thomas Gilbert, Robert Mager, Thiagi, and more. While the models they used differed, Guy’s created a synthesis that makes sense, and more importantly, works. He talked about how he refined his work to balance effectiveness with efficiency. Moreover, his approach avoids any redundant work.

Interestingly, he also recounted how his approach achieved buy-in from the stakeholders to the extent that he had to fight to not keep them all on the team through all the stages! That’s a great outcome, and it comes from demonstrating value. He focuses on where performance needs are critical, and thus it has a natural interest, but too many of the approaches can stifle that interest. Instead, his intent focus on meaningful outcomes truly engages everyone from the performers to the executives.

Guy also is quite open about the problems facing our industry. Despite the necessity of starting as order takers (essentially, “you can’t say ‘no'”), he estimates that only 20% of the time is the problem a learning or skills problem. Which resonates with other data I’ve seen about the value of training interventions! Instead, there can be many drivers for problems in performance.  His approach includes detailed analyses that identify the root cause of the problem, and when to determine that it’s worth trying an intervention. He’s quite open about how that can lead to a shift in intervention focus. At other times, it might lead to a hiatus while problems get attention.

One other thing I found interesting in the interview was how he talked about potential barriers to success up front. While it might seem like a deterrent, he pointed out how it led to making sense later. That is, folks would soon see that, for instance, supervisor support was critical to success. He includes a rigorous analysis of potential barriers as part of the book.

Quite simply, L&D has a problem of going from go-to-whoa without considering whether a course is the right solution. Guy’s book is a way to avoid doing that, and systematically evaluating what the pivotal point should be for determining whether we can successfully intervene or not, and how. There’s much more: how to manage the process, deal with stakeholders, and test your assumptions. It’s in his own inimitable style (lessons learned on editing ;), but there’s deep wisdom there. That’s my take, at least, I welcome yours.