As go the testers...

... so goes the game. Who your testers are can be a powerful determinant of how your game turns out.

Difficulty is an obvious way in which feedback can be greatly skewed by the demography of testers. Take Battle for Middle Earth: 2, for example. If all of EA's testers for the game were avid gamers and RTS fans, like me, then EA would have heard that the "Brutal" computer intelligence is too easy. But if non-gamer folks like my dad had tested the game, they might have said that "Medium" AI is hard as hell.

User interface and pacing are other elements of a game in which the demography of testers can have a huge impact.

So what about the testers of MMO betas?


NON-GAMERS
Developers like SOE and Areae have spoken of reaching out to new audiences, to folks who have never played an MMO or even an online game. That's a pretty common goal these days. But how can they get such audiences involved in the testing process? If their testers are from the usual crop of MMO gamers, then their feedback will be skewed in favor of traditional gamers.

Think about it. How are beta openings usually advertised? In my experience, it's through gaming-news sites, gaming news-magazines, and emails to veteran gamers. None of those reach the audiences they seem to be aiming for.

What about advertisements on non-gaming sites, like Yahoo's homepage, or casual sites, like MSN's game zone? Do you really think just a picture a two or three lines of text is enough to coax non-gamers into trying your MMO? I don't.

My guess?
For advertising the final product and possibly for final-phase testing, mainstream non-gamer news seems like the best bet for these developers. Somehow, their publishers need to buy interview time on TV. Forget the newspapers; they've been losing customers ever since the internet emerged onto the scene, and those who still read a newspaper nearly always watch TV news as well. Don't just place 1-minute commercials. Interviews would offer more bang for you buck, since that would be more conducive to getting the point across that your game isn't just for usual gamers.

You'll probably need to augment the usual testing methods with payed representatives; more like with single-player games. Afterall, you're not just looking for non-gamers; you're looking for non-gamers who care enough to put up with an unpolished game and care enough to offer thoughtful feedback. The fact that it's "a free game" isn't enough to convince many of these folks to aid testing. Remember when you're targetting older folks that people get more set in their ways with age. It takes more for them to pay your concept the time of day.

A more unusual way for both advertising and tester-finding would be to set up computer stations with the game in supermarkets and convenience stores. The fact that it would be such an oddity is the hook. The sight's strangeness would cause many non-gamer shoppers to be insatiably curious. Those with a little time on their hands can watch the game's representative play while that rep engages the shoppers in conversation (ideally, not all game-related) to explain and peak their interest. The rep can then hand out flyers with the website and appropriate information to interested shoppers, perhaps even with the beta codes.


GAMERS
MMOs targeted more toward existent gamers have a different challenge. How can they limit their testers to the intended audience?

Why a developer would want to limit its intended audience might not be obvious. Afterall, don't you want to reach as many people as possible?

I'll talk more in-depth about knowing your audience in my next blog, but the basic idea is that every game feature both adds and removes potential players. For some features, this is obvious (FPS-style vs turn-based, HD graphics vs more widely accessible graphics, etc.).

So, imagine that you're trying to create a Warhammer-style game, which focuses on Realm-vs-Realm warfare and grisly humor. The game might attract many gamers who are not interested in either of those features -- even many gamers who are opposed to those basic concepts -- but who join the beta and stay in beta because they're presently without any fun MMO and your game is the closest thing to fun they have. They might even be common sights on your forums, because it's someplace to voice their opinions and desires for MMOs in general. And they might love some of your game's features enough that they'll tolerate the others.

The "closet thing to fun" bit is worth repeating. Just because McDonald's is the #1 fast-food restaurant doesn't mean the majority of their customers are thrilled with the food (though, personally, I love their McNuggets, regular cheeseburgers, and chocolate shakes). A lot of people go there because, for the price and location, there's no place better. People might flock to your game's beta because it's conveniently timed and located (advertised on the right site, or a quick-and-easy download). They might even join your game's testing because a friend is also testing. If they can hang out with a friend in your game, the game itself might not have to mean much at all to the them.

Suggestion
Include in your testing sign-up form stuff like "Which of these features interests you?" and somehow automatically monitor how each person's answers match up to his or her feedback. That way, if someone like me, who enjoys more FPS-style combat, criticizes your game's turn-based combat system, you'll have some context that may or may not color my criticism. If a lot of testers call for more active combat, you'll know how many are the FPS-desirers.

It's a tricky business. Tweaking your game during the testing phase to people outside your ideal audience might actually increase your potential playerbase. You might discover that an unexpected crowd is interested in a particular feature of your game, so making a mild shift toward that crowd's usual fare results in more people buying your game. But on the other hand, a lot of your testers may be folks who are willing to play-test your game, but not willing to buy your game. God help you if you tweak your game for a portion of your testers only to finally realize they were a part of the test-only group. =/