Behind the Line: QA myths, and what makes a good tester?

Quality Assurance, most commonly referred to as just QA, is the job many people see as a great way to enter the games industry. Why wouldn’t they? It doesn’t require a specialized degree or training. Anyone can come in and get a job in QA, right? How hard can it really be? The answer: it is very difficult to do well.

As someone with extensive QA experience, as well as hiring hundreds of testers after performing over a thousand interviews, I’d like to take you through some of what is true and what is false about QA, why some of these myths persist, and what makes a truly good tester.

Interviews, like on that PSN show "The Tester"?  No, my interviews were never like this. I actually did worthwhile things.

Interviews, like on that PSN show “The Tester”? No, my interviews were never like this. I actually did worthwhile things.

 

What is a Tester

If you haven’t listened to the Quality Assurance episode of Point Streak, here is a quick run down.

A QA tester is someone who examines the functionality of a piece of pre-release software to look for defects. When those defects are found, they are reported to the development team. These reports are generally managed through a bug tracking database. Eventually, the software will be updated and returned to QA for further testing. QA must then verify that any fixes in the update actually work, that is the broken functionality now functions properly, and the change did not cause any other erroneous behavior. QA must also continue testing on this to find any more errors to report to the developers. This cycle repeats until the software is good enough to be released.

 

Common (mis)perceptions

Looking at Video Game QA from the outside, it’s easy to get the idea that testers get to “play” games all day, so it would be a totally sweet gig. Any level of critical thinking would show that isn’t the case, though. How would that kind of job justify itself?

Some go a bit further and think that testers are about making the game “fun” or providing design feedback.   This thinking stands up to scrutiny better, as that would actually serve some purpose. It’s a more pervasive idea too. Some people actually manage to get into QA and still think that this kind of feedback is in their job description.   It is not. A tester may be asked for feedback, but we’re not here to design. Feedback for “fun factor” is done with “play-testing”, or focus groups, that are carefully cultivated and set up.

One perception that is true is that QA testing in games does not require any specially trained skills. Something like a Computer Science degree can help, but it is not required. This is because most game testing is done as “black box” testing. This means that there is no visibility into the code, no engineering, no anything. You are playing the game with almost the same level of access that any other end user would have. The only special access that you would have is some debug commands to help you get through things more efficiently. Other than that, though, the tester would be interacting with the game with the same tools that a player might have sitting on their couch with their PS4. Play, look for something wrong, and report what you find.

 

Effects of (mis)perceptions as they pile up

Now, here’s where it gets insidious. While the basic premise that no special training is true, there can be people in charge that take this too far and look at ALL QA testers as completely interchangeable, or essentially as bodies to fill a seat, and that any one person off the street is just as good as any other. From that, there’s no need to view any of them as especially skilled, or even valuable. Any of them can be replaced with someone else at any time for low wages if need be.

On the other side, if testers come in to a department thinking that they get to just play games, or that they get to have a hand in designing games, they’re probably going to wind up making idiots of themselves. When the department leaders see this, it actually re-enforces the idea that these people are truly unskilled labor, and don’t deserve any better.

The vicious cycle with those two things is that when the leaders buy into the idea that these are drones that don’t need better then they don’t offer testers much in terms of pay, or consideration for quality of work environment, or most importantly any critical discernment when hiring. So, they bring on these people who will not present themselves well, in turn justifying and re-enforcing the point of view. This then goes back to the hiring, and so on.

The games industry can be pretty casual, but c'mon, let the guy work without having to worry about getting welts on his face!  PROFESSIONALISM MAN!!!

The games industry can be pretty casual, but c’mon, let the guy work without having to worry about getting welts on his face! PROFESSIONALISM MAN!!!

 

Changes over time

Every company will come to some equilibrium point. Some companies are better than others when it comes to this misperception of the value of testers. When I started, there was a very strong undercurrent of lack of respect toward QA. It was not truly disrespect, but a lack of consideration that others would get. I could also call it a lack of the positive rather than the presence of a negative.

At the time, what I suspected would happen was that as games became more and more complex, that QA would become more and more necessary, and simply throwing random people from the street at them would no longer be an effective solution. It would take quality people to do the job well, and with the right people it could even be done more efficiently for the company, and with better compensation to the testers.

To some extent, this did happen. Part of the reason is because the industry is maturing, and awareness of the value of QA is gaining traction. One engineer gets their butt saved by a bug, and they no longer sees QA as a nuisance, trying to bother them and question their work. That attitude spreads to other engineers through conversation, and eventually it seeps into the common awareness.

Another reason for improvements is the advent of the mobile game sector. This is an entirely new area that must run more efficiently than AAA console projects, but more regimented than a small independent developer, so significant QA was needed. This is a sector that got quickly populated by vets who already had the attitude shift mentioned above. It’s like starting a garden with completely fresh soil instead of having to remove all of the weeds that keep growing back. Also, new people come in and work closely with QA who are not faceless drones in another building. These QA can be long time vets actually showing the ropes of the industry to the engineers.

A third reason is that, despite all of this, QA can be a good way to get into the industry. A lot of people started, or spent time in QA. They become more senior, and the prejudices of the predecessors are, to put it bluntly, bred out over time.

 

What testing really needs

With all of this said, there are certain core truths to what is needed to be a good tester.

While no formal training is necessary to be a good tester and while some forms of education can assist in being good at testing, there are certain things that are needed to truly excel. They are, in no particular order:

Attention to detail – This is perhaps the most obvious aspect. A tester has to be able to pick up on small things as well as large things. Not only that, there are a LOT of small things to pay attention to. It isn’t just enough to, say, visit every menu. Sometimes navigating between menus is surprisingly convoluted and every way in and out of a menu needs to be explored. Sometimes information can be presented inconsistently. A function presented in one section of the game should behave the same everywhere else. Every item in the game should collide with everything else. One texture may be off center. Maybe during a cut scene, the camera moves to show an empty texture under a chair. Any of these little things that, on their own, aren’t too big a deal, but when they are allowed to add up make a game look shoddy.

Endurance, or tenacity – Tester must never look at something and JUST let it slide. It sounds straight forward, but when looking at a save flow for the fiftieth time, it becomes very difficult to maintain the same level of focus as the first. It is a trying thing that has to be experienced to be understood.

Inquisitiveness – Sometimes looking at a bug is not enough. Even if the bug can be reproduced 100% of the time, and the steps are known, there may be more going on. A tester may need to dig deeper to see if anything else is affected. Sometimes something that looks minor can actually eventually lead to a crash, or a corrupted save. Sometimes the rules of a game may sound good, but when enforced in a certain way they can cause the game to enter a state where it is impossible to progress.

Detective skills – Getting all of the relevant information for a bug is a lot like being an investigator. You have to take all of the available information to see if you can repeat what happened, but you may not know everything that went into it. Sometimes bugs can’t be reproduced regularly. Sometimes they only seem to be irregular because everything needed to make it happen again hasn’t been discovered yet. I myself once found a crash and spent hours watching a recording of it trying to figure out why it happened. Eventually I had to write up the bug as not reproducible so that the developers could be aware of it. I didn’t stop thinking about it before and after several days I realized it was because an enemy and my character tried to pick up an item simultaneously. The game then messed up and sort of gave the item to each of us. Any time QA can provide information like that to the engineers to identify the CAUSE of the bug, not just the presence of the bug, it is invaluable.

Willingness to learn – While no special training is needed, there is always something more to learn. Everything has its own quirks that need to be understood. Sometimes an issue is not with the game, but with the console, or the engine the game is using. A tester that is familiar with these things is far more effective. To get familiar with that, these details must be learned through experience. These details are rarely documented, and most likely it’s up to QA to find them in the first place. On top of that, games, as with anything in the tech sector, is constantly expanding. New platforms or other technologies are always showing up that need to be learned. Moreover, if you pay attention to the behavior of a lot of games, even without training, an understanding for how things work can develop. You may not know how to make something, but you’ll develop an understanding for how to break it.

Professionalism – This really goes as a counterpoint to the vicious cycle I described earlier. If a tester, or preferably a whole team, can maintain professional behavior, it gets a lot harder to look at the group as unskilled monkeys just there to go through the paces. That’s on top of the obvious fact that a good professional attitude really does make any work environment operate much better.

Mental flexibility – QA is just about the last in the line of people working on the game. I mean that chronologically. QA is subject to disruptions from nearly everyone else in the production pipeline. This can cause delays, bugs, changes in focus, even changes in project! Sometimes a critical fire sparks up and it has to be verified RIGHT NOW. QA will be called to drop what they’re working on now and pay attention to something else that may be completely different. If you’ve never had to do this, it’s surprisingly difficult.

Creative thinking – Games are not like enterprise software. Creators do not have the option to say: “this is a supported user path, and that is not”. In games, the players will do everything they can to get around the systems presented to them. They WILL try to break your system, so QA needs to try everything that can be thought of. There is no way to write a test plan that can encompass all possible use cases. It’s up to the creative thinking of testers to be able to take the unexpected path, combine things in unforeseen ways, or otherwise actively break the game.

 

Final thought

QA is a position that can be demanding, and exacting. It gets more respect than it did in the past, but it can still attract people who cling to the idea that it is an easy job because you don’t need a degree. In reality, you need skills that no degree will ever show that you have.

And for the love of all that is holy, QA has never been, and will never be this:

That right there is an example of people playing on the stereotypes, and in turn perpetuating them. I know people look at that for a laugh now, but for me… nnggg…  gaaahhh…

moe-rage

 


 

Kynetyk is a veteran of the games industry.  Behind the Line is written to help improve understanding of what goes on in the game development process and the business behind it.  From “What’s taking this games so long to release”, to “why are there bugs”, to “Why is this free to play” or anything else,  if there is a topic that you would like to see covered, please write in to kynetyk@enthusiacs.com

 

Leave a Reply

Your email address will not be published. Required fields are marked *