QA Mindf**k

When I read Rand’s recent post on QA I was pretty much entirely in agreement. A good QA team is a real asset to any project, especially large ones. However, a bad QA team can be a huge liability and cause problems for everyone.

Bad testers don’t understand the product they’re working on. They follow test scripts they don’t understand, write short, inaccurate bug reports and make no attempt to appreciate the context of any error.

This should all be obvious from Rand’s post. What’s maybe not so clear is what problems it causes.

What follows is not the tale of any one project but all of these things have happened at one time or another.

  • The QA team starts too early in development and doesn’t comprehend that it’s working on unfinished software. Many bug reports are raised for things that are known to be missing or incomplete.
  • The real bug reports contain so little data that they can’t be reproduced, or at least can’t be replicated without a lot of effort.
  • There are many bugs in the test scripts too, but since the QA team have no real understanding of what the software is supposed to do they refuse to take a second look at them.
  • Since QA don’t understand the desired functionality, important modules are not tested, leaving an unpleasant surprise at a later, critical stage of the project.
  • As the bugs pile up, the development team have to spend more and more time working on them instead of completing the project as originally planned.
  • Eventually, overwhelmed, the project is replanned.
  • This time the development team is under increased scrutiny. Metrics – bugs raised, time to fix, closure rate – are tracked.
  • Collecting all the metrics slows down development further.
  • Bugs pile up, the metrics start taking over the project.
  • The numbers mean everything. Few actually understand what is wrong, only how many defects there are.
  • Ironically, the pressure of developing the remaining features, working on the stream of defects and the new management overhead result in a decrease in quality as the same people are expected to do more work.
  • Management adds more people to the project to solve this problem. Unfortunately Fred Brookes wasn’t wrong.
  • The new project plan fails, further replanning…
  • Rinse. Repeat.

It’s a death spiral. Longer hours. Weekend work. Replanning. Delays. People, frustrated, leave. It’s toxic.

More often than not, projects like these get cancelled but sometimes, for political reasons, they have to be delivered. In order to do so the cycle has to be broken. This always requires someone brave to come in, slash the scope and come up with a truly realistic plan.

But it’s best just to have a constructive relationship between the development and QA teams. After all, the goal for everyone should be to deliver the best possible software.