I’ve argued before about the need for evaluation in our work. This occurs summatively, where we’re looking beyond smile sheets to actually determine the impact of our efforts. However, it also should work formatively, where we’re seeing if we’re getting closer. Yet there are some ways in which we go off track. So I want to talk about iterating and evaluating our learning initiatives.
Let’s start by talking about our design processes. The 800 lb gorilla of ADDIE has shifted from a water flow model to a more iterative approach. Yet it still brings baggage. Of late, more agile and iterative approaches have emerged, not least Michael Allen’s SAM and Megan Torrance’s LLAMA. Agile approaches, where we’re exploring, make more sense when designing for people, with their inherent complexity.
Agile approaches work on the basis of creating, basically, Minimum Viable Products, and then iterating. We evaluate each iteration. That is, we check to see what need to be improved, and what is good enough. However, when are we done?
In my workshops, when talking about iteration, I like to ask the audience this question. Frequently, the answer is “when we run out of time and money”. That’s an understandable answer, but I maintain it’s the wrong answer.
If we iterate until we run out of time and money, we don’t know that we’ve actually met our goals. As I explained about social media metrics, but applies here too, you should be iterating until you achieve the metrics you’ve set. That means you know what you’re trying to do!
Which requires, of course, that you set metrics about what your solution should achieve. That could include usability and engagement (which come before and after, respectively), but most critically ‘impact’. Is this learning initiative solving the problem we’re designing it to achieve? Which also means you need to have a discussion of why you’re building it, and how you know it’s working.
Of course, if you’re running out of time and money faster than you’re getting close to your goal, you have to decide whether to relax your standards, or apply for more resources, or abandon your work, or…but at least you’re doing so consciously. Yet this is still better than heuristically determining that three iterations is arbitrarily appropriate, for example.
I do recognize that this isn’t our current situation, and changing it isn’t easy. We’re still asked to make slide decks look good, or create a course on X, etc. Ultimately, however, our professionalism will ask us to do better. Be ready. Eventually, your CFO should care about the return on your expenditures, and it’ll be nice to have a real answer. So, iterating and evaluating should be your long term approach. Right?
Leave a Reply