Last Friday’s #GuildChat was on Agile Development.  The topic is interesting to me, because like with Design Thinking, it seems like well-known practices with a new branding. So as I did then, I’ll lay out what I see and hope others will enlighten me.

As context, during grad school I was in a research group focused on user-centered system design, which included design, processes, and more. I subsequently taught interface design (aka Human Computer Interaction or HCI) for a number of years (while continuing to research learning technology), and made a practice of advocating the best practices from HCI to the ed tech community.  What was current at the time were iterative, situated, collaborative, and participatory design processes, so I was pretty  familiar with the principles and a fan. That is, really understand the context, design and test frequently, working in teams with your customers.

Fast forward a couple of decades, and the Agile Manifesto puts a stake in the ground for software engineering. And we see a focus on releasable code, but again with principles of iteration and testing, team work, and tight customer involvement.  Michael Allen was enthused enough to use it as a spark that led to the Serious eLearning Manifesto.

That inspiration has clearly (and finally) now moved to learning design. Whether it’s Allen’s  SAM  or Ger Driesen’s  Agile Learning Manifesto, we’re seeing a call for rethinking the old waterfall model of design.  And this is a good thing (only decades late ;).  Certainly we know that working together is better than working alone (if you manage the process right ;), so the collaboration part is a win.

And we certainly need change.  The existing approaches we too often see involve a designer being given some documents, access to a SME (if lucky), and told to create a course on X.  Sure, there’re tools and templates, but they are focused on making particular interactions easier, not on ensuring better learning design. And the person works alone and does the design and development in one pass. There are likely to be review checkpoints, but there’s little testing.  There are variations on this, including perhaps an initial collaboration meeting, some SME review, or a storyboard before development commences, but too often it’s largely an independent one way flow, and  this isn’t good.

The underlying issue  is that waterfall models, where you specify the requirements in advance and then design, develop, and implement just don’t work. The problem is that the human brain is pretty much the most complex thing in existence, and when we determine a priori what will work, we don’t take into account the fact that like Heisenberg what we implement will change the system. Iterative development and testing allows the specs to change after initial experience.  Several issues arise with this, however.

For one, there’s a question about what is the right size and scope of a deliverable.  Learning experiences, while typically overwritten, do have some stricture that keeps them from having intermediately useful results. I was curious about what made sense, though to me it seemed that you could develop your final practice first as a deliverable, and then fill in with the required earlier practice, and content resources, and this seemed similar to what was offered up during the chat to my question.

The other one is scoping and budgeting the process. I often ask, when talking about game design, how to know when to stop iterating. The usual (and wrong answer) is when you run out of time or money. The  right answer would be when you’ve hit your metrics, the ones you should set before you begin that determine the parameters of a solution (and they can be consciously reconsidered as part of the process).  The typical answer, particularly for those concerned with controlling costs, is something like a heuristic choice of 3 iterations.  Drawing on some other work in software process, I’d recommend creating estimates, but then reviewing them after. In the software case, people got much better at estimates, and that could be a valuable extension.  But it shouldn’t be any more difficult to estimate, certainly with some experience, than existing methods.

Ok, so I may be a bit jaded about new brandings on what should already be good practice, but I think anything that helps us focus on developing in ways that lead to quality outcomes is a good thing.  I encourage you to work more collaboratively, develop and test more iteratively, and work on discrete chunks. Your stakeholders should  be glad you did.

 

A continued bane of my existence is the ongoing requirements that are put in place for a variety of things.  Two in particular are related and worth noting: accreditation and compliance.  The way they’re typically construed is barking mad, and we can (and need to) do better.

To start with accreditation. It sounds like a good thing: to make sure that someone issuing some sort of certification has in place the proper procedures.  And, done rightly, it would be. However, what we currently see is that, basically, the body says you have to take what the Subject Matter Expert (SME) says as the gospel. And this is problematic.

The root of the problem is that SMEs don’t have access to around 70% of what they do, as research at the University of Southern California’s Cognitive Technology group has documented. However, of course, they have access to all they ‘know’.  So it’s easy for them to say what learners should know, but not what learners  actually should be able to do.  And some experts are better than others at articulating this, but the process is opaque to this nuance.

So unless the certification process is willing to allow the issuing institution the flexibility to use a process to drill down into the actual ‘do’, you’re going to get knowledge-focused courses that don’t actually achieve important outcomes. You could do things like  incorporating those who depend on the practitioners, and/or using a replicable and grounded process with SMEs that helps them work out what the core objectives need to be; meaningful ones, ala competencies. And a shoutout to Western Governors University for somehow being accredited using competencies!

Compliance is, arguably, worse.  Somehow, the amount of  time you spend is the important determining factor. Not what you can do at the end, but instead that you’ve done  something  for an hour.  The notion that amount of time spent relates to ability at this level of granularity is outright maniacal.  Time would matter, differently for different folks, but you have to be doing the right thing, and there’s no stricture for that.   Instead, if you’ve been subjected to an hour of information, that somehow is going to change your behavior. As if.

Again, competencies would make sense.  Determine what you need them to be able to do, and then assess that. If it takes them 30 minutes, that’s OK. If it takes them 5 hours, well, it’s necessary to be compliant.

I’d like to be wrong, but I’ve seen personal instances of both of these, working with clients. I’d really like to find a point of leverage to address this.  How can we start having processes that obtain necessary skills, and then use those to determine ability, not time or arbitrary authority!  Where can we start to make this necessary change?

I had the occasion last week to attend a day of ComicCon. If you don’t know it, it is a conference about comics, but also much, much, more. It covers movies and television, games (computers and board), and more. It is also a pop culture phenomenon, where new releases are announced, analysis and discussion occur, and people dress up.   And it is huge!

I have gone to many conferences, and some are big, e.g. ATD’s ICE or Online Educa, or Learning Technology (certainly the exhibit hall).   This made the biggest of those seem like a rounding error.   It’s more like the SuperBowl.   People camp out in line to attend the best panels, and the exhibit hall is so packed that you can hardly move.   The conference itself is so big that it maxes out the San Diego Convention Center and spills out into adjoining hotels.

And that is really the lesson: something here is generating mad passion.   Such overwhelming interest that there’s a lottery for tickets! I attended once in the very early days, when it was small and cozy (as a college student), but this is something else.   I haven’t been to the Oscars, but this is bigger than what’s shown on TV.   It’s bigger than E3. Again, I haven’t seen CES since the very early days, but it can’t be much larger. And this isn’t for biz, this is for the people and their own hard earned dollars.   In designing learning, we would love to achieve such motivation.   So what’s going on?

So first, comics tap into some cultural touchstone; they appear in most (if not all) cultures that have developed mass media.   They tell ongoing stories that resonate with individuals, and drive other media including (as mentioned) movies, TV, games, and toys.   They can convey drama or comedy, and comment on the human condition with insight and heart. The best are truly works of art (oh, Bill Watterson, how could you stop?).

They use the standard methods of storytelling, strip away unnecessary details, have (even unlikely) heroes and villains, obstacles and triumphs). And they can convey powerful lessons about values and consequences.   Things we often are trying to achieve. It’s done through complex characters, compelling narratives, and stylistic artwork.   As Hilary Price (author of the comic Rhymes with Orange) told us in a panel, she’s a writer first and an artist second.

We don’t use graphic novel/comic/cartoon formats near enough in learning, and we could and should. Similarly with games, the interactive equivalent, for meaningful practice.   I fear we take ourselves too seriously, or let stakeholders keep us from truly engaging our learners. We can and should do better.  We need to understand audience engagement, and leverage that in our learning experiences.  To restate: it’s not about content, it’s about experience. Are you designing experiences?

As I push for better learning design, I’m regularly reminded that working with subject matter experts (SMEs) is critical, and problematic.   What makes SMEs has implications that are challenging but also offers a uniquely valuable perspective.    I want to review some of those challenges and opportunities in one go.

One of the artifacts about how our brain works is that we compile knowledge away.  We start off with conscious awareness of what we’re supposed to be doing, and apply it in context.  As we practice, however, our expertise becomes chunked up, and increasingly automatic. As it does so, some of the elements that are compiled away are awarenesses that are not available to conscious inspection. As Richard Clark of the Cognitive Technology Lab at USC lets us know, about 70% of what SMEs do isn’t available to their conscious mind.  Or, to put it another way, they literally can’t tell us what they do!

On the other hand, they have pretty good access to what they know. They can cite all the knowledge they have to hand. They can talk about the facts and the concepts, but not the decisions.  And, to be fair, many of them aren’t really good at the concepts, at least  not from the perspective of being able to articulate a model that is of use in the learning process.

The problem then becomes a combination of both finding a good SME, and working with them in a useful way to get meaningful objectives, to start. And while there are quite rigorous ways (e.g. Cognitive Task Analysis), in general we need more heuristic approaches.

My recommendation, grounded in Sid Meier’s statement that “good games are a series of interesting decisions” and the recognition that making better decisions are likely to be the most valuable outcome of learning, is to focus rabidly on decisions.  When SMEs start talking about “they need to know X” and “they need to know Y” is to ask leading questions like “what decisions do they need to be able to make that they don’t make know” and “how does X or Y actually lead them to make better decisions”.

Your end goal here is to winnow the knowledge away and get to the models that will make a difference to the learner’s ability to act.  And when you’re pressed by a certification body that you need to represent what the SME tells you, you may need to push back.  I even advocate anticipating what the models and decisions are likely to be, and getting the SME to criticize and improve, rather than let them start with a blank slate. This does require some smarts on the part of the designer, but when it works, it leverages the fact that it’s easier to critique than generate.

They also are potentially valuable in the ways that they recognize where learners go wrong, particularly if they train.  Most of the time, mistakes aren’t random, but are based upon some inappropriate models.  Ideally, you have access to these reliable mistakes,  and the reason why they’re made. Your SMEs should be able to help here. They should know ways in which non-experts fail.  It may be the case that some SMEs aren’t as good as others here, so again, as in ones that have access to the models, you need to be selective.

This is related to one of the two ways SMEs are your ally.  Ideally, you’re equipped with stories, great failures and great successes. These form the basis of your examples, and ideally come in the form of a story. A SME should have some examples of both that they can spin and you can use to build up an example. This may well be part of your process to get the concepts and practice down, but you need to get these case studies.

There’s one other way that SMEs can help. The fact that they are experts is based upon the fact that they somehow find the topic fascinating or rewarding enough to spend the requisite time to acquire expertise. You can, and should, tap into that. Find out what makes this particular field interesting, and use that as a way  to communicate the intrinsic interest to learners. Are they playing detective, problem-solver, or protector? What’s the appeal, and then build that into the practice stories you ask learners to engage in.

Working with SMEs isn’t easy, but it is critical. Understanding what they can do, and where they intrinsic barriers, gives you a better handle on being able to get what you need to assist learners in being able to perform.  Here are some of my tips, what have you found that works?