Learnlets

Secondary

Clark Quinn’s Learnings about Learning

Kill the curriculum?

by 5 Comments

Harold Jarche (@hjarche) retweeted his prior post on “First, we kill the curriculum“, and generated some serious interest.   For instance, Mark Oehlert (@moehlert) was inspired to write “Harold Jarche is Wicked Smart and We Need to Talk about Curriculum“.     I know Harold, and he is wicked smart (see this skewering of homework), so I commented on his blog and it seems we may have a semantics difference as opposed to a fundamental one.   Still, I want to make the point.

Harold, starting from the premise that the web is as fundamental a change as was the printing press, and, as the press could foster content, so the web can foster connections.   The emergent nature of knowledge out of a network argues against a fixed curriculum and instead for contextualized knowledge.   Arguing against a fixed curriculum, he says this:

a subject-based curriculum will always be based on the wrong subjects for some people. Without a subject-centric curriculum, teachers could choose the appropriate subject matter for their particular class

and this is, I think a valid point.   There’s too much focus on rote, and already out-dated knowledge.   Making my lad continue to demonstrate his ability to do the times tables ad nauseum only kills his love of learning.   And the first year of middle school seems to be much more about turning them into manageable prisoners rather than learning much of anything.   Things are moving so fast that it’s hard to imagine that much of what we learn, other than vocabulary, math rules, and science basics are necessary.   Jim Levin argued 30 years ago that learning multiplication and long division was outdated in the age of the calculator and that estimation was the necessary skill to ensure you were in the right ball park.   Why are we still teaching long division?   In short, drill and kill is pretty dumb.

So, is there a curriclum?   I think so, and Harold really says so too:

ensuing that students have mastered the important processes. Some of the processes that readily come to mind are critical thinking; analysing data; researching; communicating ideas; creating new things

Now, I think we’re arguing over whether skills are a curriculum, and I reckon they are, that is a focused set of learning outcomes we’re trying to achieve.   Not content, but skills.   I do believe there are some fundamentals, like levers, and gravity, and the associative property, but these are frameworks and models, tools upon which we build a flexible set of problem-solving, coupled with just the sort of skills Harold’s talking about (and I’ve talked about before).

The point is, the world’s changing, and yet we’re not equipping our kids with the necessary skills.   We need a new pedagogy, problem-focused on things kids are interested in, as Harold suggests, and focusing on their information seeking and experimentation and evaluation and the self-learning skills, not on rote exercise of skills.   I don’t do long-division anymore.   Do you?   Do you graph sentences?   Do you remember formulas?   I don’t think so.   What you do is look up information, make job aids (why are stickies so ubiquitous?), or program in the formulas or use a tool.

Whether or not we call it a curriculum, those problem-solving skills what I want my kids learning, and it’s not happening.   State standards are a joke. I don’t want them learning how to bold in Word, I want them understanding the concept of ‘styles’.   I don’t want them learning how to color a square in Powerpoint, I want them to be effective in communicating visually.   I want them to be learning how to solve ill-structured problems (cf another wicked smart person, David Jonassen)!

I don’t mind the revolutionary statement “kill the curriculum”, but I might just mean it as “kill the current curriculum”, because I do believe that the most effective path to help develop those skills is a formal learning process.   However, it’s likely to be socially constructive in nature, not instructive.   Let’s kill schooling, and reinvent schools as learning labs, with curricula focused on skills and attitudes, and perhaps a minimal core of knowledge.   Which means our standardized tests need tossing, too, but then that should also be obvious. Portfolios and contextualized abilities, not rote knowledge tests.   I reckon Harold and I (and Mark, another wicked smart person) are agreeing furiously.   Anyone for a revolution?

On the road again

by Leave a Comment

I like going to conferences: exchanging ideas, meeting new people, and just variety.   However, I haven’t been on the road since early June for any conferences, after running a workshop at ASTD’s international conference and then presenting at DAU/GMU’s Innovations in eLearning Conference.   But it’s that time again.

First, Jay Cross and I will be presenting on the Chief Meta-Learning Officer article we wrote at the Fall CLO Symposium Sept 28-30.   We’ll be riffing on the results of the survey we made available as part of the article, looking at what folks are saying about how their organization is learning.   There are big opportunities for organizations to improve how the facilitate and leverage their employee ideas, and we’re hoping to help that vision come forth.

At DevLearn (Nov 10-13), Jay and I will be running a 1 day workshop on how to be a Chief Meta-Learning Officer, and I expect we’ll capture some of the process and outcomes that led my attendees at the elearning strategy workshop to say things like “powerful and overwhelming – in a good way!”,   “Very current for today‘s priorities”, “extremely useful … I learned so much”, and “Really makes the shift from just
learning and takes it to performance.”   While it’ll focus more on the social and informal, that’s where a lot of opportunity is, and it’s really a whole shift about thinking of the learning organization’s role.

I’ll also be presenting the performance ecosystem in an abbreviated form as a concurrent session. I will also partner with Richard Clark to talk about pragmatic mobile development.   I’m looking forward to it.

I was also reflecting about what makes a good conference.   I don’t know about CLO (my first), but I love the Guild events, and I was trying to figure out why.   I think it’s because they do as good a job as anyone at making the event a good experience for all: attendees, exhibitors/sponsors, and speakers.   Others come close, but they really strike the best balance.

From an attendee perspective, you want speakers that cover the breadth and depth you want, for different levels of experience, and access to vendors without being hammered with pitches.   As a speaker, you want to maximize your exposure if you get to speak, and be treated like a valuable contribution. I can’t speak what it’s like from a vendor perspective, but I reckon it’s fair access to attendees without onerous costs or restrictions.   Somehow, the Guild events strike this balance the best, from my perspective as a speaker and attendee.

The ongoing success suggests others feel the same. I was just reviewing the speaker list, and see that in the very first timeslot, you’ve got Allison Rossett going up against Ruth Clark and Ray Jimenez, among others!   That’s a pretty heady lineup.   The topics are similarly spread to be as interesting as the speakers, with user-generated content, rapid elearning, mobile, games/simulations, and more.   And again, that’s only the first timeslot!

I’ll be online presenting in October and again in January, but for someone who’s in elearning, I still value the face to face time when I can get it.   Hope to meet you at one of these, please do introduce yourself or say hi!

The Performance Environment

by 12 Comments

I’ve represented the performance ecosystem in several ways in the past, and that process continues to occur.   In the process of writing up a proposal to do some social learning strategizing for an organization, I started thinking about it from the performer perspective.PLE

Now, personal learning environments (PLE) is not a completely new concept, and quite a number of folks contributed their PLEs here.   However, I wasn’t creating mine so much as a conceptual framework, yet it shares characteristics with many.

I realized there were some relevant dimensions, so I added those in, including whether they tend to be more reflective or active, and whether they’re formal or informal.   Note that I played a little fast and loose in the positioning to hopefully not make the connections too obscured, so it’s not quantitatively accurate so much as conceptually indicative.   Also, I’m trying to catch categories of tools, not specifics.   Still, I (apparently :) thought it was interesting enough to try to get feedback on.

So, what do you think? Am I missing a channel?   A connection?   Feedback solicited.

Design ‘debt’ and quality process

by 8 Comments

A tweet from Joshua Kerievsky (@JoshuaKerievsky) led me to the concept of design debt in programming.   The idea is (quoting from Ward Cunningham):

Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite…. The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.

I started wondering what the equivalent in learning design would be. Obviously, software design isn’t the same as learning design, though learning design could stand to benefit from what software engineers know about process and quality.   For example, the Personal Software Process‘ focus on quality review and data-driven improvement could do wonders for improving individual and team learning design.

Similarly, refactoring to remove typical bad practices in programming could easily analogize to the reliable patterns we see in Broken ID.   There are mistakes reliably made, and yet we don’t identify them nor have processes to systematically remedy them.

What are the consequences of these mistakes?   It’s clear we often take shortcuts in our learning design, and let’s be honest, we seldom go back. For big projects, we might create iterative representations (outlines, finished storyboards), and ideally we tune them once developed, but seldom do we launch, and then reengineer based upon feedback, unless it’s heinous.   Heck, we scandalously seldom even measure the outcomes with more than smile sheets!

For software engineering, the debt accrues as you continue to patch the bad code, rather than fixing it properly (paying off the principal).   In learning design, the cost is in continuing to use the bad learning design.   You’ve minimized the effectiveness, and consequently wasted the money it cost and the time of the learners.   Another way we accrue debt is transfer learning designed for one mode, e.g. F2F delivery, and then re-implement it as elearning, synchronous or asynchronous.

In software engineering, you’re supposed to design your code in small, functional units with testable inputs and outputs, and there might be different ways of accomplishing it inside, but the important component are the testable results.   Our learning equivalent would be how we address learning objectives, and of course first we have to get the objectives right, and how they build to achieve the necessary outcome, but then it shifts to getting the proper approach to meeting objectives. If we focus on the latter, it’s clear we can think about refactoring to improve the design of each component.

Frankly, our focus on process is still too much on a waterfall model that’s been debunked as an approach elsewhere.   We don’t have quality controls in a meaningful way, and we don’t check to see what reliable mistakes we’re making.   Maybe we need a quality process for design. I see standards, but I don’t see review.   We have better and better processes (e.g. Merrill’s Ripple in a Pond), but still not seeing how we bake review and quality process into it.   Seems to me we’ve still a ways to go.

Complicit Clients

by 5 Comments

I regularly rail against cookie-cutter learning design, boring elearning, etc.   I like to blame it on designers who don’t know the depths of learning behind the elements of design, and perhaps also on managers who don’t work to ensure that the learning objectives are tied closely to meaningful business outcome.   And I think that’s true, but of course there’s another culprit as well: clients who just ask for the same old thing!

I regularly work with a couple of partners who use me when there’s a need to go to the ‘next level’, whether it’s to mobile, pushing the engagement envelope, or working more strategically (that’s one of the way I help clients, too).   However, too often they’re just asked to turn content into courses, and the clients don’t care that the learning objectives in that content are too low-level, too knowledge-focused, completely abstract or de-contextualized, and generally not meaningful.   Now, my partners generally push back a bit, trying to help the client realize the value of a deeper design, but many times the client doesn’t want to put any more money in, doesn’t want to think about it, they just want that course up with a quiz (even with a pre-test!, *shudder*).   And my partners will go along, because creating elearning is their business and they can’t just turn away work.

And I’ve heard that from in-h0use departments as well.   As one of the attendees at my strategic elearning workshop a couple of months ago said, the managers from other business units say “just do that stuff you do” and don’t want any deeper thought into it.   They want it fast, based upon the content, and apparently don’t care that it isn’t going to lead to any meaningful change.   Or don’t know the difference. Hey, they learned that way, so it must be OK, right?

However, I think we owe it to the learners, to those clients, and to ourselves to start educating those clients, internal or external, about good learning.   You’ve got to know it yourself first, of course, but once you’re doing it anyway, there’s really no extra overhead at the first level.   But you want to start pushing back: “what’s the behavior that needs to change/”, or “what decisions do they need to be able to make that they can’t make correctly now?”   And, we need to ask “how will you know that it’s changed? What are the metrics that you’re trying to impact?”   Once you’ve got them thinking about measurable change, you have the opportunity to start talking about meaningful impact and good design to achieve outcomes.

Frankly, you can’t complain about relevance to the organization if you’re not fighting to achieve better outcomes, ones that matter.   So, educate yourselves, improve your processes, and then fight to be doing more meaningful stuff.   Hey, we’re supposed to be about learning, and marketing our services is really about good customer education! Get them educated, and get to be doing more meaningful and consequently rewarding design.

Guff: a conversation in 3 parts. Part 3

by Leave a Comment

A: “Ok, you‘ve got me thinking about this social learning guff.   But it sounds expensive as well as difficult.   Suppose I need a whole social media system, some big installation.   Not sure I can sell it up the chain.”

B: “One thing at a time.   First, it doesn‘t have to be expensive.   You likely already have some of the social media infrastructure, and other ways can be darn near free, but of course the rest of it does take time and effort.”

A: “Well, the cost is good news.   But I‘ve got to have payoff numbers.   Intangibles are a hard sell.”

B: “I hear you.   That‘s why it‘s worth it to take some time and do the back of the envelope numbers.   It‘s not like you can pull someone else‘s numbers off the shelf and apply them, though there are examples that can provide guidance, like the customer numbers.”

A: “Customers?   I thought this was internal”

B: “Oh, internal‘s a big opportunity, but so are conversations with customers, supply chain partners, any stakeholders that can be the source of valuable interactions.   Companies have found value crowd-sourcing new products and processes, having customer communities self-help, and even facilitating communities related to their products and services.   And, of course, there have been some spectacular mistakes by ignoring social media!   Have you heard about the cluetrain”

A: “As in ‘get a clue‘?   What is all this, crowdsourcing, cluetrain”

B: “Sorry.   Crowdsourcing is getting a lot of people to contribute ideas.   It‘s the ‘room is smarter than the smartest person in the room‘ (if you manage the process right), carried to the next level.   The Cluetrain manifesto was a marvelously foresightful and insightful recognition that with the power of the network, you no longer can control the information about your company, so you have to start having a dialog with customers.”

A: “So, we need social internal and external, eh”

B: “Yep, that‘s the idea.   And you figure out how much value you can get from your customers by having them provide you feedback, how much by making it easier to help themselves. That‘s on top of the benefits of reducing time to get answers and increasing the quality of internal ideas.”

A: “Sounds hard to quantify.”

B: “Well, it‘s not necessarily easy, but it is doable.   It just takes some time, but during that time you‘ll really be exploring the opportunities to make your company more effective.   There are big wins on the table, and it‘s kind of a shame if you ignore them or walk away.”

A: “Does this mean I can take the cost of the training department away”

B: “No, but changing it.   It‘s not replacing training, though having the social media infrastructure more effective.   Face it, most training is a waste of money not because it‘s not necessary, but because it‘s done so badly.”

A: “I‘ll say.”

B: “So why do you keep doing it”

A: “Because it‘s supposed to be important!”

B: “And it is, but if it‘s important, isn‘t it worth doing well”

A: “I suppose.”

B: “Here‘s the picture: you hire people, but they can‘t know everything they need to, you have proprietary processes, unique products, etc.   So you have some formal learning to get them up to speed, right”

A: “Yes, that‘s why we have it.”

B: “But once they‘re had formal training, they‘re not really productive until they‘ve had a chance to put those skills into play, and refine them. They become practitioners through practice. And then with enough time and guidance, they become your experts.”

A: “It‘s when they get beyond that novice stage that they‘re useful.”

B: “But that‘s when you ignore their needs, and there‘s so much more you can do. Practitioners don‘t need courses, but that‘s about all we do for them, when we should be giving them tools and resources.   Experts should collaborating, but the most we do with them typically is have them offer courses. It‘s broken.”

A: “And social media will hep with those latter two, supporting practitioners and experts.”

B: “Exactly!   And it can assist in making the formal learning better too.   But it requires expanding the responsibility of the training department to be a learning group, not removing the training department.”

A: “Isn‘t this IT?   Or maybe operations or engineering”

B: “Nope, they‘re stakeholders, but you don‘t want IT trying to facilitate conversations!”

A: “Darn right. But trainers aren‘t going to be able to do it either.”

B: “Yep, it‘s a shift, but they or at least the instructional designers should have the grounding in learning to make the shift.   It‘s a new world, and some shifts have to occur.”

A: “I‘ll say, it‘s changes for managers too.”

B: “Yep, new skills for all in learning, new roles, new ways of working. To cope with the new world in which we have to work in: faster, more agile. Eh”

A: “Got it.   Guess I‘d better get me some guff!” Grins.

Guff: a conversation in 3 parts. Part 2

by Leave a Comment

A: “Remember our discussion yesterday?   I‘m still leery of this social learning guff.   Sure, I want my folks to collaborate.   But they talk now; they‘ve got phones and email.   They can get courses if they need them.   Why do I need more”

B: “You‘re right that they‘re collaborating now.   But are they doing it efficiently?   Is what they‘re sharing accurate?   Do they go to the right people?   There‘re two problems: they probably don‘t have the best tools, and the probably don‘t have the best skills.   The evidence is that folks aren‘t doing it well.   If it‘s so critical, as you suggested yesterday, don‘t you want it optimal”

A: “Sure, but what‘s all this social media stuff got to do with it”

B: “A couple of things.   First, if someone finds an answer here, do you want someone else to have to find it again over there”

A: “Well, no.”

B: “Right.   And, if someone‘s not going to the right person, or not doing good searches, don‘t you want to help them improve”

A: “Well, yeah.   Obviously. Or kick their sorry backsides out!”

B: “Retention‘s easier than recruitment, and investing in your people‘s been shown to pay off.”

A: “You‘re right.   Ok, so you still haven‘t answered my question.”

B: “By putting in social media, we‘re providing the architecture where someone‘s answer can be shared, systematically.   Rather than leave the informal, social learning to chance, we‘re facilitating it both systemically, and personally.”

A: “Architecture, you make it sound like buildings.”

B: “Well, it is, it‘s infrastructure that supports appropriate activity.   You wouldn‘t use offices as a warehouse, and you wouldn‘t put a coffeemaker in a bathroom.   The point is to use the right tool for the job.”

A: “Great metaphor, not!”

B: “Ok, but you get the point.”

A: “So if I build it, they will learn”

B: “Of course not.   If you don‘t have a culture where it‘s safe to contribute, they won‘t.   If it‘s not safe to admit mistakes, you can‘t learn from them.   If you haven‘t established the culture, identified the skills, organized the change, or staffed appropriately, it‘s not going to happen.”

A: “Who‘s got time for that?!”

B: “Your competitors”

Guff: a conversation in 3 parts. Part 1

by 2 Comments

A: Looking up from reading.   “Guff!”

B: Curious.   “What‘s guff”

A: “All this social learning stuff.”

B: “Really, you think so”

A: “Yeah, I mean, learning‘s learning, and who needs to make a ‘social‘ out of it?   We‘ve got courses, if they want to be social in the classroom, fine, but all this hype about social learning is just a way for consultants to try to sell old soda in new bottles.”

B: “So you think learning is about courses”

A: “Sure.   What else”

B: “Well, let me defer that answer, and ask you another question.”

A: “Oh, so you‘re one of those, eh?   Answer a question with a question?   Ha.   Go ahead, shoot.”

B: “If learning‘s not important, what is”

A: “That‘s easy, nimbleness.   We‘ve got to adapt, innovate, create, we need to be faster than the rest.   Heck, they can clone a product in months, or less. You‘ve got to be agile!”

B: “So just executing isn‘t enough”

A: “Heck no!   You‘ve got to have the ‘total customer experience‘ locked down, and that means optimal execution is just the cost of entry.   Thriving is going to require continually introducing improvements: new products, new services.”

B: “OK, let‘s get back to your question, what else learning might be.”

A: “About time.”

B: “So, think about that innovating, problem-solving, creativity, etc.   That‘s not learning”

A: “No.”

B: “Do they know the answer when they start”

A: “No, or they‘d just do it.”

B: “Right. The answer is unknown, they have to find it. When they find it, have they learned something”

A: “Alright, I see your game. Yes, they‘re learning, but it‘s not like courses, it‘s not education!”

B: “Right, courses are formal learning.   That‘s the point I want to make, using the term ‘learning‘ to just talk about courses isn‘t fair to what‘s really going on.   There are informal forms of learning that are just the aspects you need to get on top of.”

A: “Oh, okay, if you want to play semantic games.”

B: “It‘s important, because this ‘social learning‘ you call guff is the key to addressing the things you‘re worrying about!   Formal learning serves a role, but there‘s so much more that an organization should be concerned about.”

A: “So here comes the pitch.”

B: “And it‘s straightforward: do you want to leave that innovation and creativity to chance, or do your best to make sure it‘s working well?   Because the evidence is that in most organizations it‘s nowhere near what it could be, and there are systematic steps to improve it.”

A: “C‘mon.   Can you tell me someone who‘s doing it well”

B: “Sure.   Just a few small firms you might‘ve heard of.   Intel‘s used a wiki to help people share knowledge.   Sun‘s capturing top performance on video and sharing it.   SAP‘s getting customers to self-help and contribute to new product ideas.”

A: “Sure, the tech companies, but how about anyone else”

B: “Caterpillar‘s got communities of practice generating ROI, Best Buy‘s getting a lot of advantage through internal idea generation, the list goes on, and those are only the ones we‘ve found.”

A: “Ok. I suppose it makes sense, but still, that label…”

B: “I hear you.”

Virtual Worlds #lrnchat

by 4 Comments

In last night’s #lrnchat, the topic was virtual worlds (VWs).   This was largely because several of the organizers had recently attended one or another of the SRI/ADL meetings on the topic, but also because one of the organizers (@KoreenOlbrish) is majorly active in the business of virtual worlds for learning through her company Tandem Learning.   It was a lively session, as always.

The first question to be addressed was whether virtual worlds had been over or underhyped.   The question isn’t one or the other, of course.   Some felt underhyped, as there’s great potential. Others thought they’d been overhyped, as there’s lots of noise, but few real examples.   Both are true, of course.   Everyone pretty much derided the presentation of powerpoints in Second Life, however (and rightly so!).

The second question explored when and where virtual worlds make sense.   Others echoed my prevailing view that VW’s are best for inherently 3D and social environments.   Some interesting nuances came in exploring the thought that that 3D doesn’t have to be our scale, but we can do micro or macro 3D explorations as well, and not just distance, but also time. Imagine exploring a slowed down, expanded version of a chemical reaction with an expert chemist!   Another good idea was for contextualized role plays.   Have to agree with that one.

Barriers were explored, and of course value propositions and technical issues ruled the day. Making the case is one problem (a Forrester report was cited that says enterprises do not yet get VWs), and the technical (and cognitive) overhead is another.   I wasn’t the only one who mentioned standards.

Another interesting challenge was the lack of experience in designing learning in such environments.   It’s still new days, I’ll suggest, and a lot of what’s being done is reproductions of other activities in the new environment (the classic problem: initial uses of new technology mirror old technology).   I suggested that we’ve principles (what good learning is and what VW affordances are) that should guide us to new applications without having to have that ‘reproduction’ stage.

I should note that having principles does not preclude new opportunities coming from experimentation, and I laud such initiatives.   I’ve opined before that it’s an extension of the principles from Engaging Learning combined with social learning, both areas I’ve experience in, so I’m hoping to find a chance to really get into it, too.

The third question explored what lessons can be learned from social media to enhance appropriate adoption of VWs.   Comments included that they needed to be more accessible and reliable, that they’ll take nurturing, and that they’ll have to be affordable.

As always, the lrnchat was lively, fun, and informative.   If you haven’t tried one, I encourage to at least take it for a trial run. It’s not for everyone, but some admitted to it being an addiction! ;)   You can find out more at the #lrnchat site.

For those who are interested in more about VWs, I want to mention that there will be a virtual world event here in Northern California September 23-24, the 3D Training, Learning, & Collaboration conference.   In addition to Koreen, people like Eilif Trondsen, & Tony O’Driscoll (who has a forthcoming book with Karl Kapp on VW learning) will be speaking,   and companies like IBM and ThinkBalm are represented, so it should be a good thing. I hope to go (and pointing to it may make that happen, full disclaimer :).   If you go, let me know!

Making designing good learning easier

by Leave a Comment

On my last post, I got a comment that really made me think.   The problem was content coming as PPTs from SMEs, and the question was poignant: “Given limited time and resources on a project how can you plan in advance to ensure that your learning is engaging and creates effective outcomes?”   I commented a reply, but I’d like to elaborate on that.

I like the focus on the ‘planning’ part: what can you do up front to increase the quality of your learning outcomes?   It’s a recursive design problem: people need to be able to design better, what training, job aids, tools, and/or social learning can we develop to make this work?   Having just done this on a project where a team I was a member of   were responsible for generating a whole curriculum around the domain, I can speak with some confidence about how to make this work.

First, are the tools.   Too often, the templates enforce rigor around having the elements, rather than about what makes those elements really work.   So, on the project, I not only guided the design of the templates, but the definitions associated with the elements that helped ensure they accomplished the necessary learning activities.   For example, it’s no good to have an introduction that doesn’t activate the relevant prior experience and knowledge, doesn’t help the learner comprehend why this learning is important, or even accomplishes this in an aversive way (can you say: “pre-test“?   :).   This is the performance support component, that helps make it easy to do things well and more difficult to do the wrong thing.   Similarly with ensuring meaningful activity in the first place, etc.

Next is the understanding.   This comes both by creating a shared understanding in the team, and then refining the process, making the outcome a ‘habit’.   First, I’d worked with some of the team before, so they shared my design principles, then I presented and co-developed with the client that understanding.   Then, as first draft content came out, I’d critique it and used that to tune the template, and the understanding amongst the content developers.

The involvement in refining the design process took some time, but really paid off as the quality of the resulting output took a steep increase and then stabilized as good quality learning experience yet reproducible in a cost-effective way and sustainable and manageable way.

As I’ve mentioned before, the nuances between bad elearning and really effective and engaging content are subtle to the untrained eye, but the outcomes are not, both subjectively from the learner’s experience, and objectively from the outcomes.   You should be collecting both those metrics, and reviewing the outcomes, as they both provide useful information about how your design is working (or not) and how to improve it.

If it matters, and it should, you really should be reviewing and tuning your processes to achieve engagement and learning outcomes.   It’s not more expensive, in the long term, though it does take more work.   But otherwise, it’s just a waste of money and that is expensive!   You’ll end up in the situation Charles Jenning’s cites, when”you might as well throw the money spent on these activities out the window.”   Don’t waste money, spend the time assuring that your learning design processes achieve what they need to.   Your organization, and your learners, will thank you.