Clark Quinn's Learnings about Learning
(The Official Quinnovation blog)

28 December 2011

Failing to Learn

Clark @ 1:04 PM

My colleague Harold Jarche pointed me to a post by Dave Snowden about deliberative practice, which I found interesting for a facet not part of the key article (which makes worthwhile points).  Among a list of important requirements for meaningful activity that is part of effective learning (i.e. it’s not just 10K hours of practice that makes an expert, but what sort of practice has an effect), Dave cites that “at least half of … experiments should fail”.  Think about that for a minute.

What that’s saying is that at least half of the money you invest in new things could be conceived of as being wasted.  You might be considered a very ineffective manager if 50% of your investments don’t yield returns!  Now, first of all, I’m sure you recognize that failed experiments aren’t a complete waste, as long as you learning something (“when you lose, don’t lose the lesson” as the saying goes).  Still, 50%  might still seem like a high failure rate.  But is low risk really good?

I remember hearing a talk by a Canadian AI researcher (who’s name escapes me after all these years) who had studied the optimal ratio of success to failures in helping a system learn. Now this was particular to the learning algorithm he’d chosen, but his result was roughly that you learned fastest if you failed two-thirds of the time, or around 67% failure.  Now that’d be pretty disheartening, but if you could take emotion out of the equation, e.g. made it safe to fail, would learning faster be a big enough argument to support bigger failure?

It depends on a lot: on how well you discern the lessons from failure, how well you tolerate failure, how much social scrutiny and how tolerant that public viewpoint is, but it’s interesting to contemplate what might be an optimal context for failure, and given that, what would be the fastest way to learn, and capitalize on that learning.  You want your experiments to be designed in the first place to yield maximum information, but if they do, what would a valuable success rate be?

I do believe that they who adapt fastest will be the survivors.  That adaptation may be subconscious, but I think conscious reflection is a valuable component.  Certainly for sharing the learning, so no one else has to make the same mistakes.  So are you learning just as fast as you can?

21 December 2011

Reflections 2011

Clark @ 5:33 PM

On this, the shortest day of the year here in the northern hemisphere, it seems like a good time to reflect on the past year and look forward to the coming one.  It’s certainly been an interesting year, and a good one personally.  I feel there are lots of reasons to hope for good things to be coming too.  The opportunities are big.

First of all, mobile really has taken off.  I know that many people are still wondering about how to take advantage of mobile, but awareness of the opportunities are growing. I’ve conducted several mobile brainstorming sessions with clients who have started with some apps and are looking to go further. I think more should be thinking about mobile from a strategy perspective as well.  While an initial initiative or two is ok, you have to recognize that mobile is a platform upon which solutions are created, and consequently the approach should go beyond just individual development efforts.  Still, the benefits of supporting workers wherever and whenever they are can make some big impacts on organizational outcomes.

Another area of excitement is social and informal learning. Even firms that have big concerns about security and government scrutiny are finding that they need to address it, and are beginning steps. Hey, if the CIA can deal with it, your organization can.  With the ITA in particular we have been successful in getting some traction in this area, and hope to help more organizations come to grips in the coming year.  The chance to really accelerate innovation and more is hard to ignore.

And I personally am excited about more activity around content: about more quality design, more structure around the content, and more awareness of the need for architectural work around it. I was thrilled to hear more discussion about this in a number of forums, and I think the opportunities to capitalize on finely granulated and articulated content will provide flexibility and personalization going forward.

Finally, it does appear more organizations are getting the message about moving to a strategic position in terms of learning technology.  And, of course, that means getting strategic about learning (big ‘L’ learning), and then aligning with the overall organization strategy. Of course, such an effort will not only result in better outcomes for the organization but also more credibility for the L&D group.

I’m really grateful to all those who’ve had a role in the past year (and in my past, period) in co-developing new understandings, and I look forward to more opportunities in the coming year.



14 December 2011

Authentic Learning

Clark @ 1:29 AM

This week, #change11 is being hosted by Jan Herrington (who I had the pleasure of meeting in West Australia many years ago; highly recommended). She’s talking about authentic learning, and has a nice matrix separating task type and setting to help characterize her approach.  It’s an important component of making our learning more effective.  On the way home from my evening yesterday, I wrote up some notes about a learning event I attended, that seem to be perfectly appropriate in this context:

I had the pleasure of viewing some project presentations from Hult Business School, courtesy of Jeff Saperstein. It’s an interesting program; very international, and somewhat non-traditional.

In this situation, the students had been given a project by a major international firm to develop recommendations for their social business. I saw five of the teams present, and it was fascinating.  I found out that they balanced the teams for diversity (students were very clearly from around the world including Europe, Asia, and Latin America), and they got some support in working together as well.

Overall, the presentations were quite well done. Despite some small variation in quality and one very unique approach to the problem, I was impressed with the coherence of the presentations and the solidity of the outcomes.  Some were very clearly uniquely innovative new ideas that would benefit the firm.

The process was good too; the firm had organized a visit to their local (world class) research center, and were available (through a limited process) for  questions. A representative of the firm heard the presentations (through Skype!) and provided live feedback. He was very good, citing all the positives and asking a few questions.

Admittedly they had some lack of experience, but when I think how I would’ve been able to perform at that age, I really recognized the quality of the outcome.

This sort of grounded practice in addressing real questions in a structured manner is a great pedagogy. The students worked together on projects that were meaningful to them both in being real and being interesting, and received meaningful feedback. You get valuable conceptual processing and meta-skills as well. The faculty told me afterward that many of these students had worked only in their home company prior to this, but after this diverse experience, they were truly globally-ready.

How are you providing meaningful learning experiences?

9 December 2011

Further (slow) thoughts on learning #change11

Clark @ 1:30 AM

I’ve been monitoring the comments on my #change11 posts, and rather than address them individually, I’m posting responses.  So, a couple of questions have recurred about the slow learning concept.  One is how the notion of quick small bites reflects a slower learning process.  Another is how it might play out in the organization.  And a final one is about the overall pedagogy.

embedded learning versus event learningTo address the first one, the notion is that the learnings are wrapped around the events in your life, not where you’re taken away from the context of your life to have a learning experience.  I think of this as embedded learning versus event learning.  Yes, it’s quick bits, but they don’t mean as much on their own as in their cumulative effect over time.  Whereas the cumulative effect of the event model dissipates quickly, the distributed model builds slowly!

To address the pedagogy, it’s about having little bits of extra information that connect to the events in your life, not separate (unless the events in your life aren’t frequent enough, and then we might create little ‘alternate reality’ events that create plausible and fun scenarios that also provide the desired practice to develop you on the path.  It’s not breaking up event-based learning into smaller chunks so much as wrapping around the meaningful events in your life, when possible.

And that pedagogy will very much be our choice.  I do hope we can take the opportunity to include a sufficient level of challenge, and the opportunity to personalize it, rather than keep it generic. Consider minimalist approaches, weave in learning-to-learn, connecting people as well as providing additional information.  For instance, we should be asking personalization questions afterward (whether via system or person).  The algorithms hopefully will have some serendipity as well as relationships to my personal experience.  Some elliptical material. This would support discovering new relationships in learning, as well, as we mine the effects of some random juxtapositions across many experiences.

How to make this practical in organizations worried about immediate productivity?  In my experience, it’s already happening. Folks are (trying) to take responsibility for their learning.  They take the social media cigarette breaks to go out and connect to their networks when the office blocks access through the firewall.  They’re discussing work topics in LinkedIn groups, and using Twitter to both track new things and to get questions answered.  The question really is whether orgs will ignore or hamper this versus facilitating it.  That’s why I’m part of the Internet Time Alliance, where we are working with organizations to help them start supporting learning, not just offering training.

We do see small bits of moves toward slow learning, but I don’t like to assume everyone’s yet capable of taking ownership of it. And, yes, the sad state of the world is that typical schooling and old-style management can squelch the love of learning and not develop the skills that are needed.  We have multiple challenges, and I’m just suggesting the concept of slow learning, a drip-irrigation versus flood metaphor, is a wedge to help drive us out of the event-based model and start addressing the issues raised: pedagogy, curricula, infrastructure, technology, politics, and more.  The efforts to build such a system, I reckon, will force progress on many fronts.  Whether it’s the best approach to do that is a separate question. I welcome your thoughts.

And a thanks to all for their participation this week, it’s been a learning experience for me as well!

7 December 2011

Making Slow Learning Concrete #change11

Clark @ 1:30 AM

It occurs to me that I’ve probably not conveyed in any concrete terms what I think the ‘slow learning’ experience might be like. And I admit that I’m talking a technology environment in the concrete instance (because I like toys).  So here are some instances:

Say you’ve a meeting with a potential client.  You’ve been working on how to more clearly articulate the solutions you offer and listening to the customer to establish whether there’s a match or not. You’ve entered the meeting into your calendar, and indicated the topic by the calendar, tags, the client, or some other way. So, shortly before the meeting, your system might send you some reminder that both reiterated the ‘message’ you’d worked out, and reminded you about pulling out the client’s issues.  Then, there might be a tool provided during the meeting (whether one you’d created, one you’d customized, or a stock one) to help capture the important elements. Afterwards, the system might provide you with a self-evaluation tool, or even connect you to a person for a chat.

Or, say, you’re walking around a new town.  Your system might regularly suggest some topics of interest, depending on  your interest showing architecture, history, or socioeconomic indicators.  You could ignore them, or follow them up.  Ideally, it’d also start connecting some dots: showing a picture from a previous trip and suggesting “Remember we saw an example of <this> architect use here?  Well, right here we have the evolution of that form; see how the arches have…”  So it’s making connections for you.  You can ignore, pursue further, or whatever. It might make a tour for you on the fly, if you wanted.  If you were interested in food, it might say: “we’ve been exploring Indian food, you apparently have no plans for dinner and there’s an Indian restaurant near here that would be a way to explore Southern Indian cuisine”.

Another situation might be watching an event, and having extra information laid on top. So instead of just watching a game, you could see additional information that is being used by the coaches to make strategic decisions: strengths and weaknesses of the opposing team in this context, intangible considerations like clock management, or the effects of wind.

And even in formal schooling, if you’re engaged in either an individual or group problem, it might well be available to provide a hint, as well as of course tools to hand.

The notion is that you might have more formal and informal goals, and the system would layer on information, augmenting your reality with extra information aligned to your interests and goals, making the world richer.  It could and would help performance in the moment, but also layer on some concepts on top.

I see this as perhaps a mobile app, that has some way of notifying you (e.g. it’s own signature ‘sound’), a way to sense context, and more. It might ask for your agreement to have a link into the task apps you use, so it has more context information, but also knows when and where you are.

This isn’t the only path to slow learning.  Ideally, it’d just be a rich offering of community-generated resources and people to connect with in the moment, but to get people ready to take advantage of that it might need some initial scaffolding.  Is this making sense?

6 December 2011

Initial reflections on Slow Learning #change11

Clark @ 1:30 AM

I have to say that I found the comments on my initial post very interesting and challenging.  A couple of themes have emerged that partly I want to elaborate on, and partly want to reflect upon.  I believe in, and do try, ‘learning out loud’, so I’ll both try to elaborate and draw myself up short.

One of the themes that emerged is the rejection of the GPS idea, and extolling a preference for getting lost (I did explicitly mention that GPS’s do not help you learn the terrain, but I’ll add that they could).  I want to unpack that a bit. One of the issues is, for me, where you want to learn, and where you don’t.  As a definitely curious individual (in both senses, I suppose :), there’s far too much of interest for me to be able to focus on it all. As a consequence, I want to decide on what I invest effort in, and what I don’t.  So, for instance, I’ve learned about keyboard shortcuts to use Word’s outline capabilities, and I can understand where others would not (tho’ I admit to wishing everyone would learn styles, as a valuable model going forward!).  So, I think we’d use a mental GPS when it’s something we don’t want to learn, and we’d want a different relationship when we do.  If we’re self-aware learners.

There’s also the issue of how the Sage on the Side would operate? What curricula are being used?  Who’s providing the content?  I will admit that while I talk a constructivist game, I do tend to err too quickly to think about designed resources.  In my own defense, I will suggest that self-learning skills aren’t broadly distributed enough yet, and particularly for novices, there are times when they don’t know what they need and why it’s important.  I am very much a fan of guided discovery learning (not the unguided discovery learning that was used as a straw-man opponent to instructivist in the Kirschner, Sweller, Clark article), but I do believe guidance is necessary until learners demonstrate self-learning capability. And I definitely think we can and should be developing that capability, but I’m not sanguine that we’re very good at it yet.

However, I’ve earlier talked about user generated curricula, and I think that’s really the way to go.   In the long term, the communities will take ownership of the paths to competency. I’m hoping, through something like Project Tin Can (horrible  name, but…), we can support defining broad paths to capability that are maintained by the community, and even self-proposed paths that are vetted by the community (says a guy who designed his own undergraduate major).

I admit that I’m trying to focus on a system that can be supporting a person in their goals with minimal human intervention. I want it to be one of the options on the table for the ‘sage’ to draw upon (e.g. the resources can be people, too), but I’m trying to make this scalable. Idealistically, I would like communities to become more other-supporting, but I think such a system would have desirable properties in terms of developing people over time.

As I think about it, I also anticipate some serendipity in the sage approach.  There’s some randomness in our cognitive architecture, and I’d like to see some similar probabilistic behavior in the system recommendations.  When we built the Intellectricity™ system, we had options ‘competing’ for the opportunity to be presented.  We also had machine learning in the background to improve the weighting.  This is where deep analytics and semantics could generate some real interesting outcomes.

And I definitely would hope that we would explicitly be working in a layer on top of the learning goals to address meta-learning or learning to learn, so we could move from providing guidance to merely suggesting resources.  So, for example, despite my time trying to improve my learning, there’s still room for improvement and I welcome any assistance I can get.  I’m mindful of Watts Humphrey’s work in software quality, where documenting your decisions and having them reviewed identified persistent mistakes, and allowed for continual improvement (resulting in large decreases in errors, much more accurate schedules, and other tangible benefits).  The approach is certainly applicable to other areas of design and work, and I think it’d be fascinating to have systemic support (because the overhead is high enough that I haven’t been able to instill it in myself ;).

Given that we’re all novices in some areas while experts in others, I think the notion of at least a bootstrap from some support makes sense to assist transition into a community of practice.  I don’t think this is fundamental opposition to rhizomes,connectivism, or anything else, but I welcome your thoughts.  I do think that the notion of a tree of trajectories may be too rigid, so I’ll have to ponder that, but I’d like to devolve that decision to the community of practice.

Going beyond just the notion of the system, I believe that we need to start thinking more broadly about our curricula, pedagogy, and more. That’s not new, but I do believe a shift from learning events to continual development is not only needed, but doable.  Given that each of us is a member of multiple communities, it may look more like a network, but that’s OK.

3 December 2011

Slow Learning – #change11

Clark @ 5:42 PM

This is a longer post launching my week in the #change11 MOOC (Massively Open Online Course).  

Our formal learning approaches too often don’t follow how our brains really work.  We have magic now; we can summon up powerful programs to do our bidding, gaze through webcams across distances, and bring anyone and anything to pretty much anywhere. Our limitations are no longer the technology, but our imaginations. The question is, what are we, and should be, doing with this technology?

I like to look at this a couple of ways. For one, I like to ask myself “what would my ideal learning situation be?”

Stop and ask yourself that.  Go ahead, I’ll wait.  And feel free to share!

For me, that would be having a personal mentor traveling with me, looking at my tasks, providing both support in the moment, and developing me slowly over time.  I talked about how we might systematize that in a post titled Sage at the Side.  I also talked about this model as Layered Learning.  That is, layering on learning across our life.

It’s part of what my colleague Harold Jarche talks about when saying “work is learning and learning is work”, the notion that as organizations start empowering workers to adapt to the increasing complexity, there will be no difference between work and learning, and we’ll have to move away from the ‘event’ model of learning and start integrating learning more closely into our activities.  We’ll need to have a closer coupling between our activities and the resources, creating what Jay Cross calls a workscape and I’ve termed the performance ecosystem.  That is, having the tools to hand, including job aids, people, and skill development, but in a more systemic way.

Think about that: how would you construct an optimal performance environment for yourself?  What would it look like?  Again, feel free to share.

Would it look like an LMS over here, training away over there, job aids scattered across portals, and social networks hierarchically structured or completely banned?  Would you have spray-and-pray (aka show up and throw up) training?  Online courses that are clicky-clicky bling-bling? Resources accessible by the way the organization is siloed?  Even the simple and well-documented matter of spaced learning is largely violated in most of the learning interventions we propagate.  In short, all of this is in conflict with how the human brain works!

Look at how how we learn naturally, before schooling (what I call the 7 C’s of natural learning). We see that we learn by being engaged in meaningful activity, and working with others.  It’s not about knowledge dump and test, but instead about coupling engaged activity with reflection.  I like Collins, Brown, & Holum’s Cognitive Apprenticeship as a model for thinking more richly about learning.  Other learning models are not static (c.f. Merrill’s trajectory through CDT to ID2 to Ripples), and I believe they’ll converge where Cognitive Apprenticeship is (albeit perhaps my slightly adulterated version thereof).  It talks about modeling, scaffolding and release, naturally incorporating social and meaningful activity into the learning process.

Taking a broader look, too many of our systems have a limited suite of solutions to choose from, and ignore a number of features that we need.  The ADDIE process assumes a course, and still doesn’t have any real support for the emotional engagement aspect. A step above is the HPT approach, which does look at the learning need and checks to see whether the solution might be a course, a job aid, realigning incentives, or some other things. However, it still doesn’t consider, really, engagement, nor does an adequate job of considering when connecting to a person is a more valuable solution than designing content.

And while Gloria Gery’s seminal work on Electronic Performance Support Systems suggested that these systems could not only provide support in the moment but also develop the learners’ understanding, I still don’t see this in any systems in practice. Even GPSs don’t help you understand the area, they just get you where you’re trying to go. So we are still missing something.

I’m really arguing for the need to come up with a broader perspective on learning.  I’ve been calling it learning experience design, but really it’s more.  It’s a combination of performance support and learning (and it’s badly in need of some branding help). The notion is a sort-of personal GPS for your knowledge work. It’s knows where you want to go (since you told it), and it knows where you are geographically and semantically (via GPS and your calendar), and as it recognizes the context it can provide not only support in the moment, but layers on learning along the way.  And I think that we don’t know really how to look at things this way yet; we don’t have design models (to think about the experience conceptually), we don’t have design processes (to go from goal to solution), and we don’t have tools (to deliver this integrated experience).  Yet the limits are not technological; we have the ability to build the systems if we can conceptualize the needed framework.

I think this framework will need to start with considering the experience design, what is the flow of information and activity that will help develop the learner (e.g. “If you get the design right, there are lots of ways to implement it”).  Then we can get into the mechanics of how to distribute the experience across devices, information, people, etc.  But this is embryonic yet, I welcome your thoughts!

Really, I’m looking to start matching our technology more closely to our brains.  Taking a  page from the slow movement (e.g. slow X, where X = food, sex, travel, …), I’m talking about slow learning, where we start distributing our learning in ways that match the ways in which our brains work: meaningfulness, activation and reactivation, not separate but wrapped around our lives, etc.

There’s lots more: addressing the epistemology of learners, mobile technologies, meta-learning & 21st C skills, and deep analytics and semantic systems, to name a few, but I think we need to start with the right conceptions.  Some of my notions of design may be too didactic, after all, and we’ll need to couple information augmentation with meaning-making to make real progress, but I think this notion of stepping back and reflecting on what we might want to achieve and where we’re currently inadequate is an initial step.

And now the initiative is over to you. I look forward to your thoughts.


Collins, A., Brown, J.S., and Holum, A. (1991).  .  Cognitive Apprenticeship: Making Thinking Visible. American Educator, Winter.

Quinn, C. (2004).  Learning at LargeEducational Technology, 44, 4, 45-49.

Quinn, C. (2009). Populating the LearnScape: e-Learning as Strategy. In M. Allen (Ed.) Michael Allen’s eLearning Annual 2009. Pfeiffer, San Francisco.

Quinn, C. (2010). Rethinking eLearning.  Learning Solutions Magazine. April.

Quinn, C. (2010). Designing for an uncertain world. Learnlets. April.

Thalheimer, W. (2006).  Spacing Learning Over Time.  Work Learning Research.

1 December 2011

Evil Design

Clark @ 8:59 AM

In the mobile ideation session I ran today for some folks, the question came up about good and bad examples of design, and subsequent events reminded me of the topic of not just bad, but evil design. What I mean is design that is crafted to return maximal outcome to the designer, not just at the expense of the user, but even to the discomfort of the user or contrary to their intentions.  Let me cite a few examples.

First, while this has been improved somewhat, the kiosks that <my usual airline> uses to check in had a big yellow ‘continue’ button that you used to indicate you were ready to move on to the next screen. And the first couple of times in an instance it was innocuous, so you got used to using the button comfortably and automatically.  But then, you’d get the opportunity to spend some extra money – nicer seats, extra miles – and the default action, signaled by the big yellow button,was to spend the money.  This could be several hundred dollars! I fortunately didn’t get trapped (I try to get to the airport early), but I wonder how many rushing travelers inadvertently *did* manage to overspend.  I think such a design takes advantage of our cognitive architecture, falling into a pattern, in unconscionable ways.

Then, today, I was driving back to O’Hare airport from the aforementioned engagement. Following my GPS instructions and the signs, I followed the route to the airport.  Now, on the way out there was a required toll, and I drove through and paid the guy (since naturally I didn’t have a pass). The signs on the way back announcing tolls didn’t look noticeably different, and so I didn’t pay too much attention. Imagine my surprise, then, to find a toll payment arrangement requiring either a pass (which of course I didn’t have) or payment by coins.  Which I also didn’t have.  And the amount was more than the outbound fee, so even if I tried to use change, I likely wouldn’t have enough (who keeps lots of change around these days?).

The cameras no doubt caught me sitting there looking around, then calling for help, then furiously driving on after giving the camera a frustrated glare. Of course there’s a fine if you don’t pay, but there’s no way to pay except through a long URL that’d be hard to get exactly right.  You have seven days, which in one sense is nice, but might cause you to put it off and forget.  Worse, when you do go online, you’d have to known to record the license plate state and number to be able to pay!  And, of course, it’s highly likely in the rush of travel that you’d forget to do this.  This seems designed for the sheer purpose to get more fees.  For example, paying online is more expensive than paying the original fee. Why can they have a person outbound, and not inbound?  It’s capitalizing on expectations and putting you in circumstances that are likely to maximize your inability to pay in the initial instance.  What’s with that?

It gets worse, by the way. If you didn’t remember (and our brains aren’t good at rote memory) that the site is singular (Xtollway.com) and instead type Xtollways.com (a reasonable and even likely mistake), you end up at a site that looks like it can help, but instead seems to have sponsored ads and looks for clicks. If you weren’t paying attention, you could end up giving your credit card to the wrong site, and still not have paid the fine!  I’m surprised such a site can exist and not be shut down!

Our cognitive architecture has some flaws, and these can be exploited by the unscrupulous (c.f. commercial gambling).  It helps to be cognizant of it. It ranges from the designing interface, to ad campaigns, and the whole way companies conduct  business (see the Cluetrain Manifesto).

This is, BTW, at least part of the reason I don’t like gamification, as many game mechanics like adding points tap into human reactions in a way to get them to do things they might not otherwise do.  This *can* be good if it gets them to do things like lose excess weight or quit smoking, but I’d rather tap into intrinsic motivation instead.

While I’m a fan of good design, and there’s a continuum to bad design, I still prefer that to evil design.  How about you?

Powered by WordPress