Good beta testing

Darren posted SUWT podcast #9 today, and it was definitely a good one. I felt the need to reply to every discussion they raised. Of course, two of my replies turned out so long that I decided to move them over here. So here's the first one, followed by the second tomorrow.

Sean made a great point about beta testers needing guidance. If you just let folks install the game and say "There, have at it!", you're going to be disappointed with the feedback (or lack thereof). Testers need to be directed and encouraged just as much as players do.

It's actually (winks at the SUWT#9 crew and listeners) pretty coincidental that the subject would come up now, since I happened to receive invitations to two betas today (I never thought that would happen).

Anyway, so here are some random thoughts on why many testers fail to test and how the situation can be improved.


FEEDBACK INTERFACES
First, you need to make it as painless and easy as possible for testers to submit feedback.

In-game tools
EQ2's beta had a great in-game bug reporting system. Testers are probably ten times more likely to submit feedback if they can do it right after the bug strikes or right after experiencing the content that needs tweaking. As I recall, SOE's tool automatically logged information like character location, too. The more automatically logged context, the better.

If you provide an in-game feedback form, allow players to select either a specific bug category (Inventory, Gear, Sound Effects, etc) or a general category (Graphics, Audio, Characters, etc). If you only provide specific categories, testers will often be unsure which to select. Aside from making feedback more complicated (thereby discouraging regular testing), this also results in feedback getting forwarded to the wrong place. There are always grey areas between categories, afterall.

Sorted forums
Just as important, SOE did a good job categorizing the forums into reports on different aspects of the game. The more focused the sub-forums, the more confident the tester is that his voice will be heard. Sub-forums generally mean that threads get pushed down (and seemingly missed/forgotten) slower.

Focused sub-forums also help testers figure out if a report has been made before. Sometimes the devs want reports to be repeated so they can get an idea of how common the problem is or how tweaks are coming along, but I'm sure there are plenty of folks like me who don't want to beat the devs over the head with known issues all the time. "Search" tools are imperfect, at best. They generally pull a lot of thread results that have little connection with the topic you were searching for, so a search tool isn't the answer. Too many forum categories can confuse the player, but smart divisions can really help.

Connect the two
If possible, allow testers to write longer and/or more generalized critiques and suggestions while in-game. The in-game tool can limit most bug reports by size. But it can also have a tab by which the player can choose categories for longer feedback, and it can forward that feedback to the appropriate sub-forum on the forum site. Class balance is an example of something that might be included on this tab.

The reasons are simple. For one, more complicated non-bug feedback, like class balance concerns, can be just as reliant on momentary experiences as quick bug reports. You want feedback while the tester's experience is fresh on his mind, such as right after a battle (when his necromancer was completed incapacitated by the cleric's Root spell, for instance). And second, again, testers are more likely to submit feedback if they can do it in-game. It's more efficient use of the tester's time, anyway.


EARLY TESTING VS FINAL TESTING
Another consideration is that it's completely different testing a game in alpha or early-beta than testing just before release. I've tested half a dozen games, and experienced both. They're very different.

For one thing, the newbie areas of the game are the first to get sorted out and polished. This happens for many reasons, two of them being that the testers are fresh to the game and that more testers have the time to pass through the early content than the endgame. This also benefits the developer because, as Damion Schubert recently noted, the early game is where hooking players is most vital; after they've invested time in the game, it's easier to make them want to play.

As a result, late-beta testers typically (hopefully) don't notice many bugs in the early areas. They have to progress further into the game to come across regular problems. So it's perhaps unfair and unreasonable to expect the same immediacy of feedback from these testers as you can expect from alpha and early-beta testers.

The final phase of testing should also be more focused on balance. The developer shouldn't be adding much (if any) new content in the final month or two. Every little addition has the potential for causing major conflict and implentation issues, and you don't want to undo much of your polish so close to release. Again, polish of the early game content is absolutely essential, because players' initial experiences will color their expectations of the entire game.

I'm sure there are other important differences, but for brevity's sake...