I was on a call yesterday with a team that had developed a stunning engine-driven scenario. It had rich complexity, good visual and interaction design, appropriate challenge, etc. This was something I’d be proud to show.
However, they’d had a focus group testing which had soundly rejected it. I was helping them try to understand was how could this happen.
What became clear was that a couple of simple, and avoidable, mistakes doomed the result. Mistakes that had little to do with the monumental task of creating the underlying model, and all to do with user perceptions.
First, the system threw you out the first mistake you made. And with no feedback about why! There were people who played it systematically to figure it out (your typical gamers), but the average audience was frustrated. This is easy to fix, by the way.
The second mistake was that they hadn’t set expectations for the focus group. The younger folks were unhappy it didn’t have the polish of a commercial game, and the older folks complained that the simulation didn’t have the depth of the real experience. Both of these could be addresssed by setting expectations up front (and the beginning information was typically dull and dry, not matching the dynamic music and going on way too long).
I suggested that with minimal work that focus group result could be turned around. A review before bringing it to the focus group would’ve caught this.
The end result was that the business decision was made to pull the plug on the project, and would regroup to think about whether to try again. It’s a shame, since they’d spent a reasonable amount of money (actually quite little for what they’d gotten), had a great core, and only needed to do a little window dressing. It’s also a lesson on just how important it is to sweat all the details.