The Benefit of User-Testing a Prototype

I’ve been wondering if it’s possible to quantify how “useful” it is to user-test a game’s playable prototype before full-scale development begins in earnest. I’m defining a “user-test” as “spending a few weeks observing and interviewing a group of twenty or thirty diverse consumers as they play your prototype.” Unfortunately, a really good analysis requires data that I doubt anyone has access to: the success rates of roughly equivalent games, that were and were not tested at the formal prototype stage, and were released into approximately equivalent market environments. Among other figures.

Given that I’m unlikely to see such numbers anytime soon, I had to settle for a simple experiment. I created a very basic excel model that calculates the average return for a game project that has been carefully tested (and potentially killed outright) at the end of prototyping, versus the average return for a game that semi-automatically proceeds from prototype to full development. A visual representation of the model (decision tree) can be viewed below:

It is important to note that in both cases (“testing” and “no testing”), you need not assume that the prototype is either greenlighted or killed, and nothing in between. The model permits you to assume that in both cases, changes to the prototype may occur. The cost of those changes is included in the total development cost. The real question is this: How much would user-testing need to increase your odds of success in order to be “worth it”?

There are several large assumptions built into my model, and several large market factors that I have ignored entirely. For example, I assume that the average next-gen title with a healthy budget will “fail” 80% to 85% of the time. Failure is defined as $8M in sales. These are obviously very, very rough guesstimates. My model does not take into account marketing costs, platform fees, inflation, team morale effects, etc, etc. I tried to include those factors, but it cluttered the model without substantially altering the outcome. Suffice to say, please don’t try to scrutinize this with a microscope. It is not intended to be perfect, or even close. It simply want to start a conversation, and perhaps get a glimmer of insight into an important question.

Anyway, onto the good stuff. My base assumptions: $400K prototype cost, $30K testing cost, $10M full dev cost, 20% chance of totally killing the project after prototype testing, 20% chance of success with the help of testing feedback, and 15% chance without (in other words, a very conservative success rate increase of just 5% with testing). Success breeds $100M in sales, Failure breeds $8M. You can change these numbers in the excel file, and the outcome values will automatically be updated. Go ahead and play with them.

My results: user-testing, on average, results in ~$1M greater sales. This figure skyrockets if you assume that testing is worth more than just a 5% increase in your odds of success. On the other hand, if you decrease the development budget and/or increase “failure case” sales by any significant percentage, testing ceases to be worthwhile. Assume a much higher base level of success (in all cases), and testing also ceases to be wortwhile.

One interesting (if obvious) takeaway: as development budgets increase, testing your prototypes on potential customers becomes an ever more worthwhile risk-limiting device. And regardless, if you believe that user-testing in relatively early stages of development could boost your chances by even just 5% or 10%, you really should consider doing it.

One response to “The Benefit of User-Testing a Prototype

  1. Cool idea. Simple enough to easily understand, yet it probably has a great deal of predictive power.

    My immediate impression was that $400k is a damn expensive prototype, and that 5% is a very modest increase in viability. This model ignores a) time and b) the fact that you can create multiple prototypes and pick the best. Of course, all of these things would only increase the attractiveness of testing if incorporated into the model.

    It’s quite jarring to think about games in this probablistic manner. You think that you’d “know it when you see it” and that no one would spend $10 million on a concept so flawed that it would be rejected in a hypothetical testing phase. But of course when you’re working on a project you lose perspective and your guesses are as good as chance. Also, decision-makers tend to be quite divorced from an understanding of the games themselves.

    So yeah! The model is simplistic, but any refinement would only strengthen your point: testing early is good for risk management.

    Has anyone ever done a compilation of how major games were tested? I imagine Blizzard, Bungie, and Nintendo do a ton of testing.

Leave a Reply

Your email address will not be published.