Working Without A Crystal Ball

Note: I think this post may be interesting to anyone creating, investing in, or distributing games (regardless of whether or not they are Xbox Live Arcade games.) However, I needed to ramble through some seemingly tangential stuff to make my point. Please bear with me. 🙂

XBLA portfolio management is a complex thing… I’m one part cat-herder, one part traffic cop, one part talent scout, and one part “quality control.” (The latter part is especially tricky… who wants to be the guy who turned down Katamari because “the art was weak”, or one of the eight publishers who turned down Harry Potter because “the writing could use polish.”) I approach these roles with a healthy dose of humility (and even anxiety), knowing that at any moment I could become “the moron who turned down [fill in the blank].” Unfortunately, the longer I hold this position, the more likely that becomes!

Trying not to be a moron

So I’ve put systems in place to hopefully help reduce the risk of my own tastes (or lack of vision) from polluting the portfolio. I can’t really discuss the details, but they include a sort of “wisdom of crowds” feedback loop, in which indie submissions are screened and rated by a group of my colleagues within Microsoft (who are asked NOT to discuss the submissions with each other before rating them — mainly to avoid group-think.) The wisdom of crowds can make my forecasts more accurate, and it can help compensate for any subconscious biases I have. Unfortunately, what I don’t believe it can do is help me identify future mega-hits (i.e. “the next Geo Wars“.)

Our marketing team can’t help with that, either. But first (for all you marketing haters out there), let the record show that XBLA’s two most experienced marketing people have regularly out-predicted the rest of the “experts” on the team (myself included) when it comes to sales of upcoming titles. Unfortunately, they stumble where everyone else does… attempting to predict the truly big winners. Were my job to simply build a portfolio of generally-successful revenue-earners, I would hand over the reins to marketing and go on extended vacation. But Microsoft needs more than that… it needs a diverse portfolio that appeals to consumers of all types, innovative content that (hopefully) inspires the industry, appealing IP, and last but not least, mega-hits that encourage consumers to buy more Xbox 360s!

Luck vs. Madden

Unfortunately, I simply don’t believe it’s possible to (reliably) predict original mega-hits, which is precisely why companies like EA have focused so strongly on franchises like Madden and Harry Potter in the past. Riding a mega-hit is far safer than trying to predict one. There are boatloads of research that back up this assertion; the most recent report that caught my eye was in my articles of interest a few months ago. A quote from that report:

Intrinsic “quality,” which we measured in terms of a song’s popularity in the independent condition, did help to explain success in the social-influence condition. When we added up downloads across all eight social-influence worlds, “good” songs had higher market share, on average, than “bad” ones. But the impact of a listener’s own reactions is easily overwhelmed by his or her reactions to others. The song “Lockdown,” by 52metro, for example, ranked 26th out of 48 in quality; yet it was the No. 1 song in one social-influence world, and 40th in another.

Music, like games, can be filtered by objective criteria that nearly everyone agrees upon (for example, I hereby bet that a song consisting of primarily nails on a chalkboard and cats screaming will never become a commercial success.) But once you get past those basic principles, marketing and public whim rule the day.

Improving your odds (or, “how to do something other than Madden”)

So, if you believe all this to be true, what’s the solution? One obvious answer: more content! Place more bets, even if it means spending less per bet. (Here’s an HBR article, focused on the research I referred to earlier, which argues for just this.)

For most web-based portals, this is a no-brainer. Pour on the content, then use various ranking and merchandising methodologies to elevate the most popular stuff. For XBLA, the issue is more complicated. As I’ve written in the past, we’re trying to foster a rapidly growing (but still emerging) ecosystem, which (in part) means keeping content flow at reasonable levels while we improve the Marketplace platform, and while our audience grows. We also have a dedicated team of people who help indies get their content onto the service, and that team can only be stretched so far. So, for the near term, we optimize on portfolio breadth and generally avoid very similar games. Of course, clones exist precisely because predicting original hits is so difficult, so avoiding them is a double-edged sword for XBLA and for developers. (Furthermore, many consumers appreciate minor variants of a game mechanic that they know and love.) In the long-term, there are several potential solutions to these challenges, but that’s a topic for another day.

What about developers?

Of course, “making many bets” is easy for XBLA. I manage a growing portfolio of many games. And thanks to XNA, I have an increasingly powerful mechanism via which to identify hits… a great community of developers who are creating and filtering them for me! See the Dream Build Play contest winners for examples.

Independent developers do not have this luxury; one or two bad calls and they may very well go bankrupt. So what should they do? One possibility is a variant of “making many bets” — quickly making many prototypes and sharing them publicly. Of course, this isn’t an original notion; many developers have bought into “quick prototype” development methodologies (though far fewer have been bold enough to exhibit their prototypes. I’ve personally been enjoying Petri Purho’s public experiments.)

By revealing their prototypes, developers can get invaluable feedback from the community and an early read on the likely success of the game. Furthermore, they can turn the community feedback loop into a buzz engine that drives tremendous hype for the game. Concerns about “giving away the farm to competitors,” while well-placed, are ultimately not worth abandoning this strategy over. After all, any veteran of this industry knows that execution is 99% of what makes a game successful. Ideas are cheap.

Wrapping up

Nobody likes to face the notion that, no matter how brilliant or creative you are, guaranteeing a hit in this industry (without established IP) is usually impossible. Well, unless you’re Blizzard, but I’m pretty sure they’re inhuman, have signed a contract in blood, or something equally otherworldly.

I face this challenge every day. My solution is to harness the wisdom of my colleagues, to increase the quantity of content on our service as quickly as possible (without jeopardizing the ecosystem), and to help foster the growth of the XNA community. These are imperfect solutions, but they’re the best available to me at this moment in time.

Publishers can also do well by placing many (small) bets across a wide area, while (as always) relying on established IP to help stabilize revenues. But independent developers need to find their own solutions. If quickly creating and sharing prototypes doesn’t work for you, find something else that does. Hoping to be “the next Blizzard” is great, but 99% of developers won’t be the next Blizzard. In the off-chance that your company falls into that 99% bucket, having a plan B could help.

8 responses to “Working Without A Crystal Ball

  1. Interesting perspective David — definitely applies to many things – art and music as well as games.
    As a developer, one thought would be to adopt rapid prototyping, then build an internal blind testing/ranking crowd to help evaluate
    what you\\\’ve got.

    That won\\\’t help with the chemistry of timing, connections, and promotion, but it would be an interesting working process.

  2. So, if you believe all this to be true, what’s the solution? One obvious answer: more content!

    Have you read Andrew Odlyzko’s Content Is Not King? The solution isn’t more content; it’s stronger relationships with consumers. Even the article by Watts that you cited argues that instead of focusing on “designing, marketing, and selling would-be hits”, marketers should “focus on creating portfolios of products that can be marketed using real-time measurement of and rapid response to consumer feedback.” Emphasis added.

    Watts doesn’t argue that you should just place more bets. He also suggests that the size of the bets should be decreased. In the context of his argument, Watts is actually recommending something much simpler: more connections to consumers. Check out “Differentiation or Salience” in the Journal of Advertising Research, Nov-Dec 1997. The authors state that “[f]amiliarity leads to liking … Brand A being ‘salient’ to more people than B is then usually also linked with whether A has, if anything, wider distribution; more shelf-space and display; more sales-people; more promotions; more word-of-mouth; more media mentions; more advertising; and probably bigger absolute profits.”

    The beauty of XBLA (and similar systems) is that there are an infinite number of points at which consumers can interact with both the Microsoft and Xbox brands. Any of those interactions can lead to more experiences with other organizations and products (attached, in some way, to Microsoft and Xbox) and any of those points can become, or serve as, what Andy Sernovitz in Word of Mouth Marketing calls “pure viral products.”

    In retrospect, I think there’s value in defining “content.” The products in a portfolio should not be viewed as content. Content should be viewed as what products can offer, and products should be viewed as points of interaction with consumers. Calling products “content” just confuses the issues.

  3. “Publishers can also do well by placing many (small) bets across a wide area, while (as always) relying on established IP to help stabilize revenues.”

    That’s called Stage-Gate. It’s an NPD technique. I don’t know why more large dev/pub outfits don’t use it, really.

    Good article, David!

  4. @bert: That definitely sounds better than simply rolling the dice, though as you say, I don’t know how much you “gain” and “lose” by passing on the initial opportunity to poll a wider audience, build buzz, etc.

    @morgan: read the sentence immediately after the one you quoted, dude. 😉 Anyway, this isn’t a beer commercial (“great taste! less filling!”) Complex problems rarely have a single, absolute solution, so I’m not sure it’s wise to position this as “content vs. relationships.”

    @adrian: Thanks for the props!

  5. Hi Dave. Interesting article and definitely a tough problem. I’m sure you get feedback a lot, but here are some of my thoughts:

    a) Open up XNA games to truly be a YouTube experience (I’m hoping this will be announced this Fall). At that point you have the masses indicating the kind of games they find interesting (to help with general portfolio decisions) let alone a way MGS can find games to truly polish. Then of course you have options like advertising dollars in the XNA/YouTube pages and turning XBLA into THE place for publishers to find fresh new talent/games.

    b) Expand from beyond peers to those in the community. Think of it as something a bit similar to eBay\’s Voices group.

    c) Take a few wildcards. The problem with asking people what they want is that they sometimes don\’t really know until they see and play it (definitely the crux of the problem you describe).

    And in terms of portfolio suggestions:

    a) Ensure a certain level of \’artistic\’ games. In general I find too little emphasis on creativity. As an example, Sony is not afraid to go avant-garde (games like flOw, Echochrome, etc). In general the few original titles on PSN seem to have more polish and show Sony’s willingness to experiment. Yes creative titles are more risky, but having critically regarded titles that may not sell well still have the side benefit of attracting publishers/developers and give a greater impression of quality of the platform to consumers (i.e. some things cannot be measured by the direct return on investment).

    b) Expand to the entire family by signing up with folks like Nickelodeon/Nick Jr.; get games based on properties like Dora the Explorer/Diego, Backyardigans,etc. They have to be more than the Flash games found on their respective websites. Downside is that a controller for kids may be needed (if the Scene It! controller does allow direction movement, that may be sufficient … or maybe even that rumoured Arcade stick).

    c) Support more \’wife-friendly\’ co-op/team play. This means games that aren’t shoot-em ups and allow for local multiplayer (a problem with things like Card games).

    Please don\’t take my comments as a negative. I’m a big fan of the Xbox 360 and XBLA. This is just some, hopefully useful, feedback.

  6. I expanded on what I wrote above here.

    Complex problems rarely have a single, absolute solution, so I’m not sure it’s wise to position this as “content vs. relationships.”

    There’s one thing that all business problems share in common: people. That’s because business is a human activity. You need people for business to be business, and you need relationships for business to occur. You can engage in commerce without content, but you can’t without people. Ultimately, all business problems can be simplified to the point where they can be solved by managing relationships. Relationships, however, are complex.

    In other words, I disagree. 🙂

  7. It seems there are two totally different issues you\’re trying to address here. One is that of maintaining a portfolio of games that keeps consumers interested in the platform and draws new ones in, and the other is that of developers choosing a worthwhile idea to pursue further. A crappy retro port that takes 2 months of a programmer\’s time and 1 month of an artist\’s time may be a great success for its developer if it sells 20,000 units, but it would probably be considered a failure with regards to making the platform more attractive to consumers. And games that do not sell well can still help the platform become more well-known and more appealing.

    There are five major aspects of the XBLA platform that make game approval a totally different ballgame than retail in terms of choosing games to draw in new customers. One is the low price of games (as well as the corresponding customer expectations), another is the instant availability, another is the competitive features (Leaderboards), another is the level of coverage/hype before a game is released, and the last one is the required Trial version of the game. All of these factors affect purchase decisions much more than they would affect retail game sales.

    I\’ve never had any trouble getting the XBLA submissions I\’ve written approved, so I\’m not going to complain about the approval process at all. What I would like to see, though, are more first party (or at least, MS-funded) efforts to fill in some of the holes in the game selection. For example, an action RPG with co-op play would be a no-brainer (no, Gauntlet doesn\’t count). Attracting new customers to the platform with flagship games that don\’t directly compete with something else on the platform helps all of us who develop for XBLA by increasing the size of the market, which, while much larger than everyone expected, is still capped out at a level that makes blockbuster a relative term.

  8. Shame I missed this post/comments earlier, a ton of fantastic information here. Without being too much of a sycophant the quality of content on XBLA has been fantastic for the next 6 months. I remember Dave when you told me that your effects wouldn’t be felt until earlier this year, and I have to say that I’m definitely happy. 2D fighters, Super Puzzle Fighter, etc. And maybe that’s the entire point.

    The nature of XBLA, and these more casual games is that you’re going to have a more discerning audience that will go towards what they know they like. Cost of entry is so low, and ease of access is so high, that I can get to the games that I know I want. As a consumer, I don’t have to make a bet. This is compounded even more by the free trials.

    As a result the likelihood of a megahit is fundamentally smaller than with retail games. With retail games there’s a grouping and flocking behavior due to the high prices and people following the crowd to hedge their bets.

    Should you really be looking at XBLA to make blockbusters? It’s unlikely to me that in its current form a single XBLA game will sell someone on a $279/$350 Xbox. It’s the entirety of the service that is compelling, and in that regards making sure the lineup is diverse and innovative is the most important thing.

    Uniqueness in innovation is extremely important as well. EchoChrome was mentioned before, and that’s precisely the kind of thing I’m talking about. PSP/PSN is the only place you’ll be able to play that game. Whereas I can get the Lumines experience on a variety of platforms. I can get the GeoWars experience on a few platforms. etc. How many uniquely innovative titles does XBLA have?

Leave a Reply

Your email address will not be published.