Testing <---> Fun

Moorgard commented today on the blurry border between beta and marketing. His post got me wondering:

• Should beta testers expect to be entertained?
• Can developers reasonably expect unpaid testers to endure boring content in order to make the game fun?

Enthusiasm
The more excited a tester is about a game, the more willing he or she will be to work, rather than just play, to help that game reach its potential. The less impressed a tester is with a beta, the more that tester will wonder if he or she made a mistake in signing up... and consider investing less work, less time, or quitting altogether.

Beta signups = blind contracts?
Certainly, disillusioned testers are still under a moral obligation to help test to one degree or another. But it can't be said that they knew full well what they were getting into, can it?

Trailers for box office films are often deceptive, despite the fact that a film trailer is the same basic type of experience as the full film (in both cases, the audience watches and listens). Interactive media, on the other hand, cannot be be comparably experienced through anything other than a demo (which is impossible for many MMOs and other games). Feature lists, FAQs, interviews, and trailers cannot ensure that the gamers who sign up for the beta test truly know that this is a game they're interested in.

Consequently, signups for beta testing are vague, largely blind, agreements. It's the equivalent of asking someone simply "Will you help me?" instead of "Will you help me to [a description of the task]?". Many testers who find that the beta doesn't match their expectations came into this position innocently (with no prior beta experience). Innocent or not, how critical can you be of someone for helping half-heartedly when that person had little knowledge of what he or she was getting into?

Timing
It seems the big question is: How far along should a game be before inviting outside testers?

I was fortunate enough to be involved in one of the early beta phases of EQ2's testing. The game was remarkably complete and functional at that time (at the early levels, at least). As a result, I was more enthusiastic in testing than usual. I reported more bugs, made more suggestions, and was basically a better tester than I was in other betas.

Are outside testers usually necessary to get the game to at least a marginally fun point in the game's evolution? I know there's much more involved than what I've covered here, but making beta fun as possible seems to be in the developer's interest.

Games masters and live events

GAME MASTERS
Even people who are not into fragfests sometimes lament the static, linear nature of PvE content caused by a game's reliance on computer, rather than human, intelligence. Game masters reintroduce the responsive creativity which only human beings are capable of. But it's important to note that there are many different ways a game master can provide content to players.

The little things
One way is for the GM to focus on creating relatively simple and brief experiences for one player or a handful of players at a time.

I've written before about using directly developer-controlled NPCs with little planning to provide what I call shadow content. This is one form of game mastering which I believe is very powerful and cost-efficient.

That Halo 3 video I linked to the other day is a great example of how one player's unique experience can catch the imaginations of thousands, exciting current players and inspiring them to continue exploring while simultaneously convincing potential players that the game can provide them with fun, memorable experiences. One video like this retains players and sells games. Ten videos like this (similarly unique experiences) convinces people that such experiences don't just happen to the very lucky; that influences the player's crucial long-term expectations (particularly vital to MMOs).

If tales of experiences like this are constantly being shared by word-of-mouth in your game and through videos on YouTube or elsewhere, then you'll have a consistently strong playerbase.

It's even more effective at player retention if you reward knowledge of such fleeting experiences. One of the great things about being a veteran of any community is being able to share stories from way back when. So, for example, a particular NPC is famous for her scowl and frightening disposition, but veteran players can remember how cheerful and kind she was before a particular incident (perhaps a whimsical scenario created by a GM with little planning, but recorded for future reference and possible use). As a result, veteran players can tout their memories, and veteran players are encouraged to interact positively with new players through the sharing of such memories (induction into the communal history -- an important element of socialization).

Full adventures
Another way GMs can provide content is through lengthier, more scripted player experiences.

These experiences would generally cost more time and assets to produce, but that is balanced by the fact that they can be repeated for multiple players. What sets these experiences apart from PvE adventures is that a good GM can slightly modify the adventure in response to particular player personalities and circumstances. They are thereby more rewarding to players and potentially more memorable experiences.

Conversation is the most obvious way this happens. A computer can only respond with rote messages and will miss many social and affective cues the players are giving. If an NPC is capable of pithy jabs and circumstantial comments, then interacting with him/her/it is infinitely more interesting.

Another way a GM can make a difference is to withhold and release information to players in response to their individual needs and preferences. Two players might both like mysteries but have vastly different prowess or skills for puzzle-solving. In that case, it makes sense to provide one with more/different clues than the other. The GM can also assess combat difficulties for players, giving a particular group an idea of how their unique combination of classes and gear will likely fare in a particular encounter, thereby helping players to find the difficulty level they're looking for.

Caution: There is a big difference between this type of GM content and shadow content. With the shadow content, any given player might expect to experience some of this content directly every now and then, but the player has no or little expectation of control over the experience (meaning both what happens and when it happens). Prescripted adventures, on the other hand, usually appear as selected services; it's like a themepark ride which the player may approach and expect a predictable experience from.

That means that, while the experience itself might not be very costly (considering that it is repeatable content), there are a number of peripheral costs. Because the player is promised a particular experience, the player may complain and perhaps expect recompense if the experience doesn't match expectations. The player may expect a certain level of accessibility, in terms of being available at regular times, in regular durations, and to equal enjoyment of all kinds of players. If you design the content to avoid these terms, then you must fight a battle of expectations.


LIVE EVENTS
Live events might or might not involve a GM, but they're generally focused at a game's entire playerbase or an otherwise large number of players.

Live events are shows. Players expect a spectacle.

You might think of the fore-mentioned GM adventures as being like going to see a movie in the theater. If the movie is bad, you'll probably be a little annoyed but not really get angry. Afterall, you probably expect movies to be hit-or-miss, you were able to join the experience by simply buying a ticket a couple minutes before the show started, and there are enough theaters around town that this one was close by. In short, you didn't plan much or invest much in the experience, and movies are common enough that one falling short isn't such a big deal.

Live events in an online game, on the other hand, are more like rock concerts. The experience usually requires more planning and investment from participants. And a good band putting on a big show near you is something that doesn't happen all the time, so you expect a truly memorable experience which you'll be telling your friends about weeks, months, or even years later. You're probably not even expecting new songs (analogous to a game's assets), but just expecting those songs to be used and augmented in a fresh and memorable way.

Live events generally require a lot of developer investment, but they can produce big rewards. These events stand tall in players' perception of your game. If the quality and polish are there, players will be talking about the experience for months. They'll excite each other about the game through memory sharing and speculation on future events. An event can even get players looking at old content in a fresh way. Live events are also more likely than regular gameplay to attract the attention of non-players. That's partially because participants are more likely to talk about fresh and big experiences; and it's partially because big events are more conducive to advertising -- it's easier for a themepark to attract people by advertising a new ride than by selling a new vision of the old park. Live events help keep the game fresh and feeling like a living game.

Live events also offer unique opportunities for testing and selling content. At rock concerts, multiple bands play together. It's common for attendees to go there for one band and "discover" one or more of the bands also performing. The supporting bands will thereby get feedback, publicity, and even sell a few albums. Likewise, a live event can include samples of future content. By doing this in the content's prototype stage (albeit, with more polish than a prototype would usually receive), developers can get feedback before more significant investment. Devs can also include samples of content from expansion packs, thereby marketing the expansions. If your game involves microtransactions, then the event can highlight some of those, like bands selling albums and paraphenalia at their concerts.


Anyway. A long article, I know. In summary:

---Small, unsolicited GM experiences are a shoe-in. Absolutely, they are well worth the expense.

---Scripted, solicited GM services can be costly. Manage expectations carefully. Ensure the quality of your GMs (patience, on-the-spot creativity, thorough understanding of both the game and social interaction, etc) and provide appealing avenues of feedback.

---Big events are great, but don't include them unless the quality is there. If the quality and polish are there, then good control of expectations can actually make these high-expense productions into low-risk scenarios (like rock concerts). They can be great testing and marketing tools. Just remember that games are about interactivity, so players should feel like participants and not just spectators.

Good beta testing

Darren posted SUWT podcast #9 today, and it was definitely a good one. I felt the need to reply to every discussion they raised. Of course, two of my replies turned out so long that I decided to move them over here. So here's the first one, followed by the second tomorrow.

Sean made a great point about beta testers needing guidance. If you just let folks install the game and say "There, have at it!", you're going to be disappointed with the feedback (or lack thereof). Testers need to be directed and encouraged just as much as players do.

It's actually (winks at the SUWT#9 crew and listeners) pretty coincidental that the subject would come up now, since I happened to receive invitations to two betas today (I never thought that would happen).

Anyway, so here are some random thoughts on why many testers fail to test and how the situation can be improved.


FEEDBACK INTERFACES
First, you need to make it as painless and easy as possible for testers to submit feedback.

In-game tools
EQ2's beta had a great in-game bug reporting system. Testers are probably ten times more likely to submit feedback if they can do it right after the bug strikes or right after experiencing the content that needs tweaking. As I recall, SOE's tool automatically logged information like character location, too. The more automatically logged context, the better.

If you provide an in-game feedback form, allow players to select either a specific bug category (Inventory, Gear, Sound Effects, etc) or a general category (Graphics, Audio, Characters, etc). If you only provide specific categories, testers will often be unsure which to select. Aside from making feedback more complicated (thereby discouraging regular testing), this also results in feedback getting forwarded to the wrong place. There are always grey areas between categories, afterall.

Sorted forums
Just as important, SOE did a good job categorizing the forums into reports on different aspects of the game. The more focused the sub-forums, the more confident the tester is that his voice will be heard. Sub-forums generally mean that threads get pushed down (and seemingly missed/forgotten) slower.

Focused sub-forums also help testers figure out if a report has been made before. Sometimes the devs want reports to be repeated so they can get an idea of how common the problem is or how tweaks are coming along, but I'm sure there are plenty of folks like me who don't want to beat the devs over the head with known issues all the time. "Search" tools are imperfect, at best. They generally pull a lot of thread results that have little connection with the topic you were searching for, so a search tool isn't the answer. Too many forum categories can confuse the player, but smart divisions can really help.

Connect the two
If possible, allow testers to write longer and/or more generalized critiques and suggestions while in-game. The in-game tool can limit most bug reports by size. But it can also have a tab by which the player can choose categories for longer feedback, and it can forward that feedback to the appropriate sub-forum on the forum site. Class balance is an example of something that might be included on this tab.

The reasons are simple. For one, more complicated non-bug feedback, like class balance concerns, can be just as reliant on momentary experiences as quick bug reports. You want feedback while the tester's experience is fresh on his mind, such as right after a battle (when his necromancer was completed incapacitated by the cleric's Root spell, for instance). And second, again, testers are more likely to submit feedback if they can do it in-game. It's more efficient use of the tester's time, anyway.


EARLY TESTING VS FINAL TESTING
Another consideration is that it's completely different testing a game in alpha or early-beta than testing just before release. I've tested half a dozen games, and experienced both. They're very different.

For one thing, the newbie areas of the game are the first to get sorted out and polished. This happens for many reasons, two of them being that the testers are fresh to the game and that more testers have the time to pass through the early content than the endgame. This also benefits the developer because, as Damion Schubert recently noted, the early game is where hooking players is most vital; after they've invested time in the game, it's easier to make them want to play.

As a result, late-beta testers typically (hopefully) don't notice many bugs in the early areas. They have to progress further into the game to come across regular problems. So it's perhaps unfair and unreasonable to expect the same immediacy of feedback from these testers as you can expect from alpha and early-beta testers.

The final phase of testing should also be more focused on balance. The developer shouldn't be adding much (if any) new content in the final month or two. Every little addition has the potential for causing major conflict and implentation issues, and you don't want to undo much of your polish so close to release. Again, polish of the early game content is absolutely essential, because players' initial experiences will color their expectations of the entire game.

I'm sure there are other important differences, but for brevity's sake...

As go the testers...

... so goes the game. Who your testers are can be a powerful determinant of how your game turns out.

Difficulty is an obvious way in which feedback can be greatly skewed by the demography of testers. Take Battle for Middle Earth: 2, for example. If all of EA's testers for the game were avid gamers and RTS fans, like me, then EA would have heard that the "Brutal" computer intelligence is too easy. But if non-gamer folks like my dad had tested the game, they might have said that "Medium" AI is hard as hell.

User interface and pacing are other elements of a game in which the demography of testers can have a huge impact.

So what about the testers of MMO betas?


NON-GAMERS
Developers like SOE and Areae have spoken of reaching out to new audiences, to folks who have never played an MMO or even an online game. That's a pretty common goal these days. But how can they get such audiences involved in the testing process? If their testers are from the usual crop of MMO gamers, then their feedback will be skewed in favor of traditional gamers.

Think about it. How are beta openings usually advertised? In my experience, it's through gaming-news sites, gaming news-magazines, and emails to veteran gamers. None of those reach the audiences they seem to be aiming for.

What about advertisements on non-gaming sites, like Yahoo's homepage, or casual sites, like MSN's game zone? Do you really think just a picture a two or three lines of text is enough to coax non-gamers into trying your MMO? I don't.

My guess?
For advertising the final product and possibly for final-phase testing, mainstream non-gamer news seems like the best bet for these developers. Somehow, their publishers need to buy interview time on TV. Forget the newspapers; they've been losing customers ever since the internet emerged onto the scene, and those who still read a newspaper nearly always watch TV news as well. Don't just place 1-minute commercials. Interviews would offer more bang for you buck, since that would be more conducive to getting the point across that your game isn't just for usual gamers.

You'll probably need to augment the usual testing methods with payed representatives; more like with single-player games. Afterall, you're not just looking for non-gamers; you're looking for non-gamers who care enough to put up with an unpolished game and care enough to offer thoughtful feedback. The fact that it's "a free game" isn't enough to convince many of these folks to aid testing. Remember when you're targetting older folks that people get more set in their ways with age. It takes more for them to pay your concept the time of day.

A more unusual way for both advertising and tester-finding would be to set up computer stations with the game in supermarkets and convenience stores. The fact that it would be such an oddity is the hook. The sight's strangeness would cause many non-gamer shoppers to be insatiably curious. Those with a little time on their hands can watch the game's representative play while that rep engages the shoppers in conversation (ideally, not all game-related) to explain and peak their interest. The rep can then hand out flyers with the website and appropriate information to interested shoppers, perhaps even with the beta codes.


GAMERS
MMOs targeted more toward existent gamers have a different challenge. How can they limit their testers to the intended audience?

Why a developer would want to limit its intended audience might not be obvious. Afterall, don't you want to reach as many people as possible?

I'll talk more in-depth about knowing your audience in my next blog, but the basic idea is that every game feature both adds and removes potential players. For some features, this is obvious (FPS-style vs turn-based, HD graphics vs more widely accessible graphics, etc.).

So, imagine that you're trying to create a Warhammer-style game, which focuses on Realm-vs-Realm warfare and grisly humor. The game might attract many gamers who are not interested in either of those features -- even many gamers who are opposed to those basic concepts -- but who join the beta and stay in beta because they're presently without any fun MMO and your game is the closest thing to fun they have. They might even be common sights on your forums, because it's someplace to voice their opinions and desires for MMOs in general. And they might love some of your game's features enough that they'll tolerate the others.

The "closet thing to fun" bit is worth repeating. Just because McDonald's is the #1 fast-food restaurant doesn't mean the majority of their customers are thrilled with the food (though, personally, I love their McNuggets, regular cheeseburgers, and chocolate shakes). A lot of people go there because, for the price and location, there's no place better. People might flock to your game's beta because it's conveniently timed and located (advertised on the right site, or a quick-and-easy download). They might even join your game's testing because a friend is also testing. If they can hang out with a friend in your game, the game itself might not have to mean much at all to the them.

Suggestion
Include in your testing sign-up form stuff like "Which of these features interests you?" and somehow automatically monitor how each person's answers match up to his or her feedback. That way, if someone like me, who enjoys more FPS-style combat, criticizes your game's turn-based combat system, you'll have some context that may or may not color my criticism. If a lot of testers call for more active combat, you'll know how many are the FPS-desirers.

It's a tricky business. Tweaking your game during the testing phase to people outside your ideal audience might actually increase your potential playerbase. You might discover that an unexpected crowd is interested in a particular feature of your game, so making a mild shift toward that crowd's usual fare results in more people buying your game. But on the other hand, a lot of your testers may be folks who are willing to play-test your game, but not willing to buy your game. God help you if you tweak your game for a portion of your testers only to finally realize they were a part of the test-only group. =/