For that matter, helping an overconfident learner find out what they don’t know they don’t know might be useful from a motivation side ;).

Great feedback, thanks!

We’ve tried to move to pre-test as an optional tailoring mechanism that also provides the opportunity to test-out of the course activities.

I’m hesitant to support pre and post test delta comparisons as a reasonable measure of effectiveness. If the existence of this measure didn’t become the primary / only checkbox that indicated evaluation had taken place, then I’d be less apprehensive. Improvement reflected in short term recall is one measure, but it’s not all measures. And in many cases it is the least important measure of effectiveness and impact, if it is important at all.

I also lean towards divorcing folks from the traditional notion of a test. Some folks really get it and build assessments and challenges based on problems and authentic context. Sadly, most do not. For pre and post tests, trivial pursuit is an annoyance for the learner and a waste of time for the person assembling the ‘test’. Temporarily changing the lexicon until we get a stronger contingent of exemplars seems to make alot of sense to me.

Skill gauges, challenges, and gating activities seem like powerful alternatives to test. It’s semantics… I know. But there’s plenty of rewiring to be done to weed out the bad habits and bad examples.

We built a self-paced for the USMC on heavy machine guns. This was for folks that were incidental gunners (you are a supply guy, your gunner is taken out of action and you are in the $h^t, what do you do now?) What a fun course to work on.

We were required to include a post test as an aggregate measure. But we did some cool things with the sections and activities. These were context and problem based, and they required 100% mastery to continue. Most of it was procedural and the activities were chunked relatively well (according to our evaluation data). We combined what we viewed as meaningful interactions that required placement of parts in the right proximity (drag and drop) but immediately chained into expanding pie menu’s that evaluated choices (place this part notch forward or notch back, rotate it, rotate it how far, what are you thinking / concerned about when you perform this step). Each step provided feedback and the procedure chains also provided an after action report that told what learner got right and wrong. They could also play back the proper procedure when they were finished. These would have been great to offer at the beginning of the section (see how well you do without any instruction).

Lots of opportunity.

@Steve: I agree. In Australia, it is recommended that you use a minimum of three assessment methods. I like to think of it as:

1) Can they do it? (skill assessment: eg. observation, demonstration & explanation, inspection of a finished product, etc)
2) Do they know why they have to do it that way? (theory assessment: eg. short answer, verbal explanation)
3) Can they think on their feet if something unexpected happens? (cognitive assesment: role-play, simulation, case study, etc)

And we’re also expected to assess whether they actually apply on it on job or not(Workplace assessment, stats analysis, supervisor review, etc).

2. When should we teach it?
—————————–
Learning for the first time (here’s everything we think you should learn) vs. learning as a result of coaching and development (here is what you should learn/brush up on)

3. How should we teach it?
————————–
Courses (do module 1 then module 2 etc) vs. learning experiences (work on this realistic project, acquire target skills, have just-in-time teaching as support if you’re stuck)

—

I think there is a difference between a pre-test and failure before you learn.

Pre-test is the top down, ‘this is what we know you need to know to do X’ and should be based on competencies identified. Fail-to-learn is more about the experience itself – ‘what can I be challenged to do that I can safely fail at doing so my brain knows it needs to learn something new?’ Then cue just-in-time teaching.

Pre-test are about skills being the focus of the teaching. This is true for stakeholder. Accomplishing a realistic goal/mission should be the focus of the learner and failure and learning and recovery is simply part of achieving that mission.

—

Apollo 13.

Pre-test ‘Can you fly to the moon and back?’ > Course ‘here’s how to fly to the moon …’

Fail-to-learn ‘Pretend you’re flying to the moon, doing everything right and then – Houston, we have a problem – the blah blah breaks and you need to build a new one using some tape and a radio and a cup – or you die’. Here’s some just-in-time teaching on how to do that.

So bring the two together – build failure into the course and treat that as the pre-test. There are no pre-tests in life, only failure, teaching, recovery from failure (I’d like to see my scores on the marriage and parenting pre-test – I might not have met the pre-requisites) :)

I too, would love to see some experimental investigation. I worry that interpretation of the phenomenon might tend to propel things that are just not that useful. I love that you are making the distinction between pre-test questions and problems / challenges. I think there is an appreciable difference between the cognition involved in answering a trivia question and applying what you know to make a decision to solve a problem.