Lessons from the trenches - How to test video games

Dinowaurs is our first fully-featured game, and through all the missed deadlines, critical errors and small successes we’ve learned an immense amount about game development. I’d like to sit here on my stoop and tell everyone how pro we were throughout the development of Dinowaurs, but the truth is we weren’t. Dinowaurs was a gauntlet that tested our talent, persistence and passion for making really fun games. Each obstacle taught us something, the hard way. But I’m of the opinion that sometimes that’s the only way to learn. But only sometimes, perhaps if you’re reading this and you want to learn how to make games, some of these lessons will rub off on you.

While it’d make sense to begin at the beginning, either with prototyping or planning your cycles, I’m not interested in making sense at the moment. At least chronological sense. So let’s begin at the end, which is where we are right now (sort of) for Dinowaurs. The testing phase.

Test early

Most of the time testing is intrinsically tied to the almighty “bug report” and while that remains to be the bulk of the focus for testers, it is also extremely important in regards to usability, gameplay and general feedback. Take it from the guys in the big leagues. Valve gets people playing the game as soon as they have a prototype so they can adjust a multitude of things before it becomes a huge mammoth of a program that requires hours to make a simple change like a color. Although, if you planned properly, a color to something like your UI should be a simple change.

But, believe it or not. We’re not Valve. Not by a mile, and most developers are in the same boat. We have limited time, money and human resources and unless we want to be out of work in 4 months, the game needs to get done within a tight schedule. So we do our best, and have mini-testing sessions when we can afford them.

For our early gameplay/usability tests we took a page from <a href=”http://2dboy.com/2007/11/12/rons-rules-for-playtesting/”>2D Boy</a> and sat down a few different types of people, some gamers, some non-gamers and watched them play. This helped us plan the direction of our UI, our tutorials and game mechanics in general. Gamers didn’t have much of a problem at all with our game, which makes sense since we’re all gamers. We design for ourselves. But non-gamers were confused in some areas and tended to need more hand holding. While we argued about this before the testing occurred with me claiming “I just want to play the damn game, i hate tutorials.” But after the testing session, it was undeniably clear that we needed to rethink tutorials. Scientific results have a way of sobering that unbridled indie-badassery very quickly.

Usability

Usability is important yet easy to overlook when working on a smaller game. You might think “Seriously, if I can figure it out, and my friend can figure it out, then I don’t care if your Grandma can’t figure it out.” Well, ok. Grandma may never understand the glorious nuances of your game’s design, but don’t you want your game played by the most people possible? If you don’t then that’s fine, but don’t expect to make a living like this. I recommend Silverback if you’re looking to get some detailed results. We didn’t hear about it until after Dinowaurs was late in beta, but from the looks of it, it should help quite a bit if for nothing else than organizing all your testing data into a nice neat package.

Work hard and work smart - Like Dr. Gregory House

Now onto the bugs. The big nasty beetly creatures that permeate any piece of software. I hate them. Any developer hates them. Sometimes they’re easy to squash and you get kind of a high from their putrid death fumes. But more often, they mock you. Wake you up in the middle of the night whilst you cry “Gomez!!!! You bastard!” But it gets easier…

Software is predictable, in a minuscule program finding the problem is a matter of looking at the few lines that make it up. But in large programs problems become much harder to find, if not improbable. Simply searching through code does not cut it. That’s where the almighty testers come in. For volunteers, we’re happy to just get bug reports from them, even though sometimes that’s tough to coax out of them. Volunteer testers are really there just to play a game earlier than the others. Nothing wrong with that, but don’t expect them to be fastidious testers along with it. Some of the volunteers will be seriously dedicated though, and want to help you test the game for bugs. These guys are awesome.

That’s the volunteers. In our case, we couldn’t pay testers, so basically Josh and I were our paid testers. In the beginning we just banged our heads against the game, hoping something would come out, but that was about as smart as banging our heads against a game. Through many hours testing became much more about the scientific method than anything else. In one epic with the fabled “No Terrain Bug” we spent 2 straight days playing hundreds of subsequent matches trying to pinpoint this bug. As with anything, it was what nobody expected. If a player used a strike weapon in a prior game, the following game would be infected with the bug. It’s something that we never would have found if we didn’t document each game, our behavior, their relationship with each other and the exact circumstances the games were played under.

To test properly and efficiently you have to steel yourself against getting sucked into the gameplay, and look at it objectively as an experiment each and every time. This is why you have to work hard (test many, many games) and work smart (use the proven scientific method of deduction). Like I hinted at in the subhead here. Finding a bug in a complex program or game is similar to diagnosing a sick patient. House even occasionally forces a patient to get sick, or test by taking them off pain killers. This is all valid. By changing one variable in “the system” you can uncover new data with which to bring to the table. He takes a purely objective view of the patient in no-holds-barred effort to give the patient the treatment they need. Clearly it seems barbaric because it’s a human being in the show, but for us, it’s just a game. Poke and prod all you want!

All in all, it’s about knowing. Once you know the exact problem it can usually be recreated locally (within a strict testing environment) and the rest of the fix is pretty much just about following the trail. The easy part. I can’t stress enough how important quality testing can be and how much time it can save your whole team. The more attention to detail you pay while testing and the better notes you take, the better you’ll get at the this skill and be able to uncover and isolate bugs much faster.

The rewards

Because Josh and I were testing so vigorously late in the development process, this freed up Mike to continue implementation of features, test and fix more minor known bugs and etc. While testing is tedious and it’s perhaps the one thing in game development I hate most, delivering a complete bug report to Mike was satisfying, and he was incredibly grateful. He could take one look at the report after we were done, recreate the bug with 100% accuracy based on our instructions for recreation and fix the bug within a matter of minutes.

It’s often the case that the worst and scariest bugs are also the smallest. The big bulky bugs are easy to find, but tend to take a bit longer to fix. The small bugs are incredibly hard to find, and it’s possible that they may never be found, or at least that’s the feeling when they come up. But once they are found, the hard part is over and we can all pump our fists in celebration.

Don’t dismiss the value of paid testers

If you’re interested in testing video games for a living, or as a way in through the ground floor of a development studio, I’d recommend this post as advice for rising to the top of your testing community fast. We value our hardcore volunteer testers for Dinowaurs on Kongregate, and if we had the resources, I wouldn’t flinch in offering them a wage position, or perhaps even a salaried position here at intuition. It would be a dream to have someone solely dedicated to finding and pinpointing bugs for us since it would free up tons of hours of our time for development of the games themselves.

All in all, this is just a snippet of the lessons we have learned throughout Dinowaurs. We will be writing up more post-mortem/soap box type stuff as we have time, or it pops up in our heads to do so. I hope this helped one of you out there.