I was reading Joel on Software the other day and he talked about a rumor that one of the reasons that Windows Vista release was extremely inconsistent and unpolished was that the testing was fully automated (or left brain). There was no traditional (whole brain) testing where the tester not only runs the software through test scripts but also looks for overall inconsistencies and usability issues. The ability to look at the complete picture is a right brain task that is hard or impossible to automate with current technology.
Now there is nothing wrong with automating testing with scripts so long as there is also human testing for how easy the software is to use. After all at the end of the day if humans don’t like using your software then you have a big problem no matter how many automated QA tests have passed. And with Vista most reviewers didn’t like using it and recommended staying with XP for as long as you can.
So if you are testing software please get some good QA folks to give you overall feedback. Or even better find some representative users to check it out. Otherwise some of those small “fit and finish” issues might cause your release to be a disaster…
