Learnlets

Secondary

Clark Quinn’s Learnings about Learning

Extending Engagement

by 1 Comment

My post on why ‘engagement’ should be added to effective and efficient led to some discussion on LinkedIn. In particular, some questions were asked that I thought I should reflect on.  So here are my responses to the issue of how to ‘monetize’ engagement, and how it relates to the effectiveness of learning.

So the first issue was how to justify the extra investment engagement would entail. It was an assumption that it  would take extra investment, but I believe it will. Here’s why. To make a learning experience engaging, you need some additional things: knowing why this is of interest and  relevance  to practitioners, and putting that into the introduction, examples, and practice.  With practice, that’s going to come with only a marginal overhead. More importantly, that is part of also making it more effective. There  is some additional information needed, and more careful design, and that  certainly is more than most of what’s being done now. (Even if it should be.)

So why would you put in this extra effort?  What are the benefits? As the article suggested, the payoffs are several:

  • First, learners know more intrinsically why they should pay attention. This means they’ll pay more attention, and the learning will be more effective. And that’s valuable, because it should increase the outcomes of the learning.
  • Second, the practice is distributed across more intriguing contexts. This means that the practice will have higher motivation.  When they’re performing, they’re motivated because it  matters. If we have more motivation in the learning practice, it’s closer to the performance context, so we’re making the transfer gap smaller. Again, this will make the learning more effective.
  • Third, that if you unpack the meaningfulness of the examples, you’ll make the underlying thinking easier to assimilate. The examples are comprehended better, and that leads to more effectiveness.

If learning’s a probabilistic game (and it is), and you increase the likelihood of it sticking, you’re increasing the return on your investment. If the margin to do it right is less than the value of the improvement in the learning, that’s a business case. And I’ll suggest that these steps are part of making learning effective,  period. So it’s really going from a low likelihood of transfer – 20-30% say – to effective learning – maybe 70-80%.  Yes, I’m making these numbers up, but…

This is really all part of going from information dump & knowledge test to elaborated examples and contextualized practice.  So that’s really not about engagement, it’s about effectiveness. And a lot of what’s done under the banner of ‘rapid elearning’ is ineffective.  It may be engaging, but it isn’t leading to new skills.

Which is the other issue: a claim that engagement doesn’t equal better learning. And in general I agree (see: activity doesn’t mean effectiveness in a social media tool). It depends on what you mean by engagement; I don’t mean trivialized scores equalling more activity. I mean fundamental cognitive engagement: ‘hard fun’, not just  fun.  Intrinsic relevance. Not marketing flare, but real value add.

Hopefully this helps!  I really want to convince you that you want deep learning design if you care about the outcomes.  (And if you don’t, why are you bothering? ;).  It goes to effectiveness, and requires addressing engagement. I’ll also suggest that while it  does affect efficiency,  it does so in marginal ways compared to substantial increases in impact.  And that strikes me as the type of step one  should be taking. Agreed?

 

My policies

by 2 Comments

Like most of you, I get a lot of requests for a lot of things. Too many, really. So I’ve had to put in policies to be able to cope.  I like to provide  a response (I feel it’s important to communicate the underlying rationale), so I have stock blurbs that I cut and paste (with an occasional edit for a specific context).  I don’t want to repeat them here, but instead I want to be clear about why certain types of actions are going to get certain types of response. Consider this a public service announcement.

So, I get a  lot of requests to link on LinkedIn, and I’m happy to, with a caveat. First, you should have some clear relationship to learning technology. Or be willing to explain why you want to link. I use LinkedIn for business connections, so I’m linked to lots of people I don’t even know, but they’re in our field.

I ask those not in learntech  why they want to link. Some do respond, and often have a real reason (shifting to this field, their title masks a real role), and I’m glad I asked.  Other times it’s the ‘Nigerian Prince’ or equivalent. And those will get reported. Recently, it’s new folk who claim they just want to connect to someone with experience. Er, no.  Read this blog, instead. I also have a special message to those in learntech with biz dev/sales/etc roles; I’ll link, but if they pitch me, they’ll get summarily unlinked (and I do).

And I likely won’t link to you on Facebook.  That’s personal. Friends and family. Try LinkedIn instead.

I get lots of emails, particularly from elearning or tech development firms, offering to have a conversation about their services.  I’m sorry, but don’t you realize, with all the time I’ve been in the field, that I have ‘goto’ partners? And I don’t do biz dev: develop contracts and outsource production. As Donald H Taylor so aptly puts it,  you haven’t established a sufficient relationship to justify offering me anything.

Then, I get email with announcements of new moves and the like.  Apparently, with an expectation that I’ll blog it.  WTH?  Somehow, people think this blog is for PR.  No, as it says quite clearly at the top of the page, this is for  my  learnings about learning.  I let them know that I pay attention to what comes through my social media channels, not what comes unsolicited.  I also ask what list they got my name from, so I can squelch it. And sometimes they have!

I used to get a lot of offers to either receive or write blog posts. (This had died down, but has resurrected recently.)    For marketing links, obviously. I don’t want your posts; see the above: my learnings!   And I won’t write for you for free. Hey, that’s a service.  See below.

And I get calls with folks offering me a place at their event.  They’re pretty easy to detect: they ask about would I like to have access to  a specific audience,…  I have learned to quickly ask if it’s a pay to play.  It always is, and I have to explain that that’s not how I market myself.  Maybe I’m wrong, but I see that working for big firms with trained sales folks, not me. I already have my marketing channels. And I speak and write as a service!

I similarly get a lot of emails that let me know about a new product and invite me to view it and give my opinion.  NO!  First, I could spend my whole day with these. Second, and more importantly, my opinion is  valuable!  It’s the basis of 35+ years of work at the cutting edge of learning and technology. And you want it for free?  As if.  Let’s talk some real evaluation, as an engagement.  I’ve done that, and can for you.

As I’ve explained many times, my principles are simple: I talk ideas for free; I help someone personally for drinks/dinner; if someone’s making a quid, I get a cut.  And everyone seems fine with that, once I explain it. I occasionally get taken advantage of, but I try to make it only once for each way (fool me…).    But the number of people who seem to think that I should speak/write/consult for free continues to boggle my mind.  Exposure?  I think you’re overvaluing your platform.

Look, I think there’s sufficient evidence that I’m very good at what I do. If you want to refine your learning design processes, take your L&D strategy into the 21st century, and generally align what you do with how we think, work, and learn, let’s talk.  Let’s see if there’s a viable benefit to you that’s a fair return for me. Lots of folks have found that to be the case.  I’ll even offer the first conversation free, but let’s make sure there’s a clear two-way relationship on the table and explore it.  Fair enough?

 

 

Ethics and AI

by 1 Comment

I had the opportunity to attend a special event pondering the ethical issues that surround Artificial Intelligence (AI).  Hosted by the Institute for the Future, we gathered in groups beforehand to generate questions that were used in a subsequent session. Vint Cerf, co-developer of the TCP/IP protocol that enabled the internet, currently at Google, responded to the questions.  Quite the heady experience!

The questions were quite varied. Our group looked at Values and Responsibilities. I asked whether that was for the developers or the AI itself. Our conclusion was that it had to be the developers first. We also considered what else has been done in technology ethics (e.g. diseases, nuclear weapons), and what is unique to AI.  A respondent mentioned an EU initiative to register all internet AIs; I didn’t have the chance to ask about policing and consequences.  Those strike me as concomitant issues!

One of the unique areas was ‘agency’, the ability for AI to  act.  This led to a discussion for a need to have oversight on AI decisions. However, I suggested that humans, if the AI was mostly right, would fatigue. So we pondered: could an AI monitor another AI?  I also thought that there’s evidence that consciousness is emergent, and so we’d need to keep the AIs from communicating. It was pointed out that the genie is already out of the bottle, with chatbots online. Vint suggests that our brain is layered pattern-matchers, so maybe consciousness is just the topmost layer.

One recourse is transparency, but it needs to be rigorous. Blockchain’s distributed transparency could be a model. Of course, one of the problems is that we can’t even explain our own cognition in all instances (we make stories that don’t always correlate with the evidence of what we do). And with machine learning, we may be making stories about what the system is using to analyze behaviors and make decisions, but it may not correlate.

Similarly, machine learning is very dependent on the training set. If we don’t pick the right inputs, we might miss some factors that would be important to incorporate in making answers.  Even if we have the right inputs, but don’t have a good training set of good and bad outcomes, we get biased decisions. It’s been said that what people are good at is crossing the silos, whereas the machines tend to be good in narrow domains. This is another argument for oversight.

The notion of agency also brought up the issue of decisions.  Vint inquired why we were so lazy in making decisions. He argued that we’re making systems we no longer understand!  I didn’t get the chance to answer that decision-making is cognitively taxing.   As a consequence, we often work to avoid it.  Moreover, some of us are interested in X, so are willing to invest the effort to learn it, while others are interested in Y. So it may not be reasonable to expect everyone to invest in every decision.  Also, our lives get more complex; when I grew up, you just  had phone and TV, now you need to worry about internet, and cable, and mobile carriers, and smart homes, and…  So it’s not hard to see why we want to abrogate responsibility when we can!  But when can we, and when do we need to be careful?

Of course, one of the issues is about AI taking jobs.  Cerf stated that nnovation takes jobs, and generates jobs as well. However, the problem is that those who lose the jobs aren’t necessarily capable of taking the new ones.  Which brought up an increasing need for learning to learn, as the key ability for people. Which I support, of course.

The overall problem is that there isn’t a central agreement on what ethics a system should embody, if we  could do it. We currently have different cultures with different values. Could we find agreement when some might have different view of what, say, acceptable surveillance would be? Is there some core set of values that are required for a society to ‘get along’?  However, that might vary by society.

At the end, there were two takeaways.  For one, the question is whether AI can helps us help ourselves!  And the recommendation is that we should continue to reflect and share our thoughts. This is my contribution.

Reflection ‘out loud’

by Leave a Comment

I am a fan of Show Your Work and Work Out Loud, but I’m wondering about whether they could mislead.  Not that that’s the intent, of course, but they don’t necessarily include reflection, a critical component. I believe they care about it, but the phrase don’t implicitly require annotating your thoughts. And I think it’s important.

The original phrase that resonated for me was ‘narrate your work’, which to me was more than just showing it.  When teachers told you to show your work, they just wanted intermediate steps. But Alan Schoenfeld’s research has documented that’s valuable to show the thinking  behind the steps. What’s your rationale for taking this step?

Teachers would be able to identify where you went wrong, but that doesn’t necessarily say  why you went wrong. On things as simple as multi-column subtraction, the answer would tell you whether they borrowed wrong or reversed the number or other specific misconceptions that would reveal themselves in the result. But on more complex problems, the intermediate steps may not preserve the rationale.

The design rationale approach emerged on complex projects for just this reason. New people could question earlier decisions and if they weren’t documented, you’d revisit them unnecessarily.  It’s important to capture not only the decision, but the criteria used, the others considered, etc.  It is this thinking  about what drove decisions that helps people understand your thinking and thereby improve it.

I don’t really think “reflection  out loud” is the right term. I like ‘narrate your work’ or ‘share your thinking’ perhaps better.  And I  do believe that those talking about working out loud and sharing your work do intend this, it’s just that too often I’ve seen people take the surface implications of a phrase and skip the real importance (*cough* Kirkpatrick’s levels *cough*). So, worst case, I’ve confused the terminology space, but hopefully I’ve also helped illuminate the valuable underpinning.  And practiced what I’m preaching ;).  Your thoughts?

Jay Cross Memorial Award 2017: Marcia Conner

by 2 Comments

The Internet Time Alliance Jay Cross Memorial Award is presented to a workplace learning professional who has contributed in positive ways to the field of Real Learning and is reflective of Jay‘s lifetime of work. Recipients champion workplace and social learning practices inside their organization and/or on the wider stage. They share their work in public and often challenge conventional wisdom. The Jay Cross Memorial Award is given to professionals who continuously welcome challenges at the cutting edge of their expertise and are convincing and effective advocates of a humanistic approach to workplace learning and performance.

We announce the award on 5 July, Jay‘s birthday. Following his death in November 2015, the partners of the Internet Time Alliance (Jane Hart, Harold Jarche, Charles Jennings, and myself) resolved to continue Jay‘s work. Jay Cross was a deep thinker and a man of many talents, never resting on his past accomplishments, and this award is one way to keep pushing our professional fields and industries to find new and better ways to learn and work.

Marcia Conner, recipient of the Jay Cross Memorial Award

The Internet Time Alliance Jay Cross Memorial Award for 2017 is presented to Marcia Conner. Marcia was an early leader in the movement for individual and social learning, and an innovator. As a Senior Manager at Microsoft, she developed new training practices and wrote an accessible white paper on the deeper aspects of learning design. She subsequently was the Information Futurist at PeopleSoft.  She also served as a co-founder and editor at Learnativity, an early online magazine.

Marcia  co-organized and co-hosted the Creating a Learning Culture conference at the Darden School of Business at the University of Virginia, leading to a book of the same title.  As an advocate for the power of learning, alone and together, she wrote Learn More Now and co-wrote The New Social Learning (now in it‘s second edition) with Tony Bingham of the Association for Talent Development. She also was the instigator who organized the team for the twitter chat #lrnchat, which continues to this day.

Marcia‘s a recognized leader, writing for Fast Company, and keynoting conferences around the world. She currently helps organizations go beyond their current approaches, changing their culture.  She‘s also in the process of moving her focus beyond organizations, to society. In her words, “I‘m in pursuit of meaningful progress, with good faith and honesty, girded by what I know we are capable of doing right now. When we assemble all that is going on at the edges of culture, technology, and (dare I say) business, we find a wildly hopeful view of the future. People doing extraordinary things, on a human scale, that has the potential to change everything for the better.”

Marcia was a friend of Jay‘s for many years (including organizing the creation of his Wikipedia page), and we‘re proud to recognize her contributions.

Helen Blunden was the inaugural award winner in 2016.

FocusOn Learning reflections

by Leave a Comment

If you follow this blog (and you should :), it was pretty obvious that I was at the FocusOn Learning conference in San Diego last week (previous 2 posts were mindmaps of the keynotes). And it was fun as always.  Here are my reflections on what happened a bit more, as an exercise in meta-learning.

There were three themes to the conference: mobile, games, and video.  I’m pretty active in the first two (two books on the former, one on the latter), and the last is related to things I care and talk about.  The focus led to some interesting outcomes: some folks were very interested in just one of the topics, while others were looking a bit more broadly.  Whether that’s good or not depends on your perspective, I guess.

Mobile was present, happily, and continues to evolve.  People are still talking about courses on a phone, but more folks were talking about extending the learning.  Some of it was pretty dumb – just content or flash cards as learning augmentation – but there were interesting applications. Importantly, there was a growing awareness about performance support as a sensible approach.  It’s nice to see the field mature.

For games, there were positive and negative signs.  The good news is that games are being more fully understood in terms of their role in learning, e.g. deep practice.  The bad news is that there’s still a lot of interest in gamification without a concomitant awareness of the important distinctions. Tarting up drill-and-kill with PBL (points, badges, and leaderboards; the new acronym apparently)  isn’t worth significant interest!  We know how to drill things that must be, but our focus  should be on intrinsic interest.

As a side note, the demise of Flash has left us without a good game development environment. Flash is both a development environment and a delivery platform. As a development environment  Flash had a low learning threshold, and yet could be used to build complex games.  As a delivery platform, however, it’s woefully insecure (so much so that it’s been proscribed in most browsers). The fact that Adobe couldn’t be bothered to generate acceptable HTML5 out of the development environment, and let it languish, leaves the market open for another accessible tool. And Unity or Unreal provide good support (as I understand it), but still require coding.  So we’re not at an easily accessible place. Oh, for HyperCard!

Most of the video interest was either in technical issues (how to get quality and/or on the cheap), but a lot of interest was also in interactive video. I think branching video is a real powerful learning environment for contextualized decision making.  As a consequence the advent of tools that make it easier is to be lauded. An interesting session with the wise Joe Ganci (@elearningjoe) and a GoAnimate guy talked about when to use video versus animation, which largely seemed to reflect my view (confirmation bias ;) that it’s about whether you want more context (video) or concept (animation). Of course, it was also about the cost of production and the need for fidelity (video more than animation in both cases).

There was a lot of interest in VR, which crossed over between video and games.  Which is interesting because it’s not inherently tied to games or video!  In short,  it’s a delivery technology.  You can do branching scenarios, full game engine delivery, or just video in VR. The visuals can be generated as video or from digital models. There was some awareness, e.g. fun was made of the idea of presenting powerpoint in VR (just like 2nd Life ;).

I did an ecosystem presentation that contextualized all three (video, games, mobile) in the bigger picture, and also drew upon their cognitive and then L&D roles. I also deconstructed the game Fluxx (a really fun game with an interesting ‘twist’). Overall, it was a good conference (and nice to be in San Diego, one of my ‘homes’).

Nathalie Nahai #FocusOnLearn Keynote Mindmap

by Leave a Comment

Nathalie Nahai opened the second day of the FocuOn Learning conference. In a rapid fire presentation, she covered 7 principles that engage individuals into behaviors. With clear examples from familiar online experiences, she portrayed how these things work. Admirably, she finished with a call to ethical behavior.

Keynote mindmap

Liza Donnelly #FocusOnLearn Keynote Mindmap

by Leave a Comment

Liza Donnelly opened the FocusOn Learn conference with stories from her career as a cartoonist. With a very personal and compelling story illustrated by her cartoons and some live drawing, she unpacked creativity and innovation. With lessons about commitment and meaning, it was a really nice kickoff to the event.

Keynote Mindmap

A learning meta-story

by Leave a Comment

Been thinking about how to generate meaningful learning in optimal (read: concise but effective) ways. And a lot of what I’ve been thinking about involves contextualized meaningful practice (no surprise there, eh?).  So how might this play out?  Thought I’d use a story to convey the experience I’m thinking of:

Pat logs on to the system, and notes that it’s time to take a crack at the next assignment.  In it is a setup  with a role for Pat to play.  The story details a business situation: the organization, it’s current status, and a situation that’s occurred that requires an action.  The details are exaggerated, so it’s a dire situation with a lot riding on the outcome. The instructions are phrased in the form of an email directly from the CEO, with pointers to some folks to talk to for assistance.

The necessity is for Pat to create a plan to address the need.  In this case, it’s a marketing plan for a new product that has been the focus of most of the organization’s effort.  With old products facing receding sales, this product  has to succeed.  The existing plan, legacy of a departed individual, is ‘old school’ and an up-to-date approach is needed.  The indicated need  is heavily aligned with this week’s topic of social-media marketing.

Pat starts work to create a document to send to the CEO. This includes  making ‘calls’ (viewing videos of quick messages from the various roles involved including the product manager, the financial officer) to find out the  parameters which are in play and to get expert knowledge.  There are also some marketing materials available.  

In  previous assignments there were support tools about creating documents and about marketing plans, but this time  such  support isn’t available.  Pat realizes  that this being a more advanced cut through the topic, it’s time to start taking ownership of the process.  The CEO has  asked for an interim plan report  before creating the entire marketing plan, and  Pat uses previous materials and adapts them to  create the  plan.

Pat will get feedback from the CEO to incorporate in the plan before putting together the final submission.  Ultimately, the success of the plan will be presented, and then feedback on the details of Pat’s submission.  The document creation will be  evaluated separately and in the context of previous documents required across this particular topic and previous ones, while the marketing plan itself will be evaluated in terms of it’s response to the context.  

Several things to note here. The contextualized performance requirement isn’t unique, of course.  This very much draws upon similar work seen in Roger Schank’s Story-Centered Curriculum and Goal-Based Scenarios. It differs in that subsequent assignments might use totally separate story settings.  It’s similar also to work like Bransford, et al’s Anchored Instruction.  The notion of embedding performance in context reflects research that shows abstract instruction doesn’t transfer as well. My own proposal (research, anyone?) is that the story should complete before the conceptual feedback is presented, or indeed that the story outcome includes the conceptual feedback in an intrinsic way.

The second important thing is that the document creation details are assessed separately, and tracked across other such assignments that might appear anywhere. The point is to develop meta-skills like digital document creation (and others such as presentations, working in groups, research, etc) as well as the domain skills.

I believe that we need learners to create complex work products that are challenging to auto-mark, because the outcomes are necessary.  This means that you need people in the learning loop; totally asynchronous isn’t going to work to develop rich capabilities. I’m trying to figure out ways to approximate that with as little human intervention as possible because pragmatically we have more learning to achieve than we have resources to achieve that (at least until we get our priorities right ;).

 

Grappling with Groups

by 4 Comments

I’m a fan of the power of social learning. When people get together (and the process is managed right), the outcomes of a negotiated understanding can be powerful.  However, in designing learning, working in groups  has some real negative perceptions  and  realities. The open question is: what to do?

The problems are well-known. As my kids complained, on group projects some team members will reliably slack, letting the most driven student do the work.  Even with a commitment, there can be differences in working style: getting started early versus preferring to do it under pressure.

Some things have been tried. When I assigned group projects, I told my students I expected them to do equal work, and would grade accordingly. If it didn’t end up being the case, they were to each write up a report on what each team member did, including themselves. Others require this, regardless, and that sounds like  a smart way to make concrete a requirement for contribution.

One  thing to be addressed  is invigilation. Is the work being tracked  in any way?  If they’re working in a collaborative environment that tracks contributions via versioning or some other way, then there’s a trail of work that can be scrutinized. Extra work, to be sure, but it’d serve as a tie-breaker if there was some question about contribution.

Another  issue is support  for working in groups. When I first assigned group work, it became clear that they didn’t know how (?!?!).  So I wrote up a little guide to doing group work, and those problems subsided.  Working together is a skill that shouldn’t be taken for granted. There should be some explicit statement of expectations if you can’t determine whether there’s reliable prior experience. (Certainly, it seems that the teachers weren’t providing guidance or oversight, in the case of my kids.)

As an aside: make sure the students know  why you’re asking them to work in groups. I’ve learned that learners will be much more willing to undertake what you assign if you explain the rationale that justifies your choice!

Then there’s them question  of  just  when group work makes sense. Given that the value-added benefit is the negotiated understanding, it would make sense to do that when the material is complex, and there’s a risk of an individual taking a unique, incomplete, and or imperfect understanding. At times when you want to assess an individual’s ability to deliver, you wouldn’t want a group project!

There’s also the  issue of the  nature of the task.  Are you just having them come to a shared understanding in representing their thinking (e.g. a response to a question) or actually produce a work product of some sort (a video, presentation, report, etc).  If you can get what you need with less effort, you shouldn’t assign a more complex project.

Which brings up the issue of the scope of the work. I would expect that the more imposing the total amount of work is, the more it would invoke those with time or effort concerns to be lulled to the lazy side.  Keeping the scope small might contribute to a greater willingness to participate.

Breaking up the deliverables is one way to manage student effort. If you have interim deliverables, it helps manage the process  and the time.  Certainly, early in a curriculum, you could provide this scaffolding (and make it explicit), and then gradually hand off responsibility for the learners to internalize  the self-management. (Meta-learning!)

Breaking it up can also manage to address the contribution. If individual submissions are required before group ones, you can at least have the learners having had to contribute  thought before sharing and creating a greater understanding.

Finally, there’s the issue of group work in an independent schedule. In a cohort model (scheduled timetable) it’s easy, but otherwise, how do you do it?  If there’s ‘critical mass’, you can have learners arrange to meet with anyone available. If there’re more, you could even have them indicate working style preferences: quick, early, what media channels. Otherwise, it’s more challenging (or a non-issue, just don’t do it).

There are lots of issues and potential solutions for addressing group work.  I can’t say I’ve found an easy solution, despite having wrestled with it. I think it’s important, so I’m curious what you’ve tried and found out!