Search This Blog


Wednesday, April 9, 2008

What GUI Test Automation Is Not

Continuing from my previous posts on GUI test Automation, today I would like to talk about what it is not.
  1. It is not capture and replay
  2. It is not meant for programming dummy
  3. It is not unit tests and shouldn't be used as such.
  4. It cannot be as effective as other form of testing, such as unit testing and manual tests
  5. The time to recover the investment is longer than other forms of testing
  6. It is not a substitution for a proper test plan.
It is not capture and replay
Automated GUI tool providers would like you to believe that their tools are extremely user-friendly and do not require you to tweak the generated scripts. While capture and replay utility can save you a lot of time because it can save you the trouble of having to lookup object names when you write your scripts, relying on it exclusively will kill your test automation effort completely, I am serious.

However intelligent an GUI test tool is, it cannot replace human judgments. It can't identify meaningful object names; it can't structure the test code so that they are readable to human eyes; it can't identify duplicated or redundant routines. It cannot do everything human can do. Without editing, the outputs of capture and replay utility are very ugly to read and even uglier to maintain.

It is not meant for programming dummy
The users of test tools cannot be programmer dummies. In fact, GUI test automation requires so much software development effort that the script writers should be properly called developer-testers.

Capture and replay utility helps to reduce the initial keystrokes, but for the scripts to be useful, they must be maintained. Maintaining these scripts are no different from maintaining your development code.

Without programming expertise, the testers will flee from automation effort as soon as the zeal died off. There is no hope for the whole test automation regime to take off.

It is not Unit testing
Unit testing and GUI test automation looks so similar at the surface. Both serve the purpose of catching regressions. But beneath the similarity, unit testing and GUI test automation are actually very different.

Unit testing should be fast, and it should be done by programmers who write the production code. Instead of playing defensive, one can turn unit testing into an aggressive force in development software by applying the techniques of TDD. Unit testing verifies the correctness of a module at atomic level.

But GUI testing is inherently slow. There is no hope of letting GUI test automation drives the software design. Instead of concentrating on lower level detail, GUI test automation seeks to verify that the program performs as expected at the top level.

It cannot be as effective as other forms of testing
Due to various constraints, the effectiveness of automated GUI testing is severely limited. So it's unfair to demand it to find bugs immediately. Automated testing is not as efficient as manual testing, because human testers are vastly more creative than computers. Human testers know how to break the software, but GUI test scripts can only do what they are told to do.

If GUI test automation is not very effective in catching bugs, then what is it good for? In my opinion, GUI test automation is very useful in proving the program works as expected as a whole. This, in itself, is more than enough to justify its own existence. Imagine if you don't have automated GUI tests, you have to perform a lot of mundane tests with your keyboard and mouse. They are boring, and they are not likely to break. But you can't skip them anyway because if they do break you will be embarrassed. If you have test automation to take over such a job then you will be relieved of the pain to run mundane yet crucial tests.

You can invest your precious time and energy into something else that are more meaningful.

Time for recoup is longer
Like any other software development effort, there is no such thing as a short-term speculator as far as GUI test automation is concerned. To reap the maximum benefits out of the test regime, you need to patiently grow your test suites. The benefits are not immediate; they are manifested over time as you achieve greater test coverage.

When we started unit testing, the benefits were immediately obvious. We managed to catch lots of regression bugs at the initial stage of our implementation and that boosted up our morale and justify the unit testing practice in no time. The developers could go home sleep soundly knowing that their new implementation didn't break existing features. For GUI test automation, the time for recoup is longer. But over the time the ability of the test suits to catch bugs does increased The GUI test regime uncovered a lot of subtle bugs that testers would otherwise overlook. Even though the numbers couldn't match those caught by unit testing or other forms of testing, still, the very fact that it discovered bugs is very significant to us, for without it we might never discover those bugs in the first place.

It is not a substitution for proper test planning!
Finally, as good as GUI test automation can get, it is not a substitute for a proper test planning. Do not think that test automation is a panacea to your testing problem. It's not. If you are lacking time in testing your application, or you simply don't know how to test your application, then GUI test automation cannot help in this manner.

Automated testing can test your application more quickly, but it can't decide what to test or what not to test. Your automated testing regime is only as strong as the weakest spot in your testing regime.


Magnus said...

Great post! I've worked as developer for almost 10 years and tried some GUI testing tools. Hated them all ;-). Too complicated, some obscure scripting language, unmaintainable tests etc. I've finally found a tool that fits my needs. Test Automation FX, which is integrated into the IDE I use (VS2008) It produces code that I understand (C#), and the test is separated from UI mapping logic which makes it easier to maintain when the UI changes. You can read more about here


Anonymous said...

I wish my managers would read this article. They all seem to think that GUI automation is an answer to improved testing efficiency and therefore reduce costs and time. GUI automation is indeed a deep discpline, and requires investment in time and people who can develop the correct, re-usuable framework. We do a lot of tranlsation/globalisation testing which involves testing GUI for many different languages for translation/globalisation defects. My managers think, if we can record a scripts in English and re-run it on other langauges, we save time and money...sadly it really is not that way, and automated GUI testing, in my view will never trump over good old human end user testing