Before I start my rant I would like to make a difference between Automated GUI Testing and unit testing. There is a lot of differences between the two. Unit test tests against a component; its scope is often limited to class libraries. But GUI testing is more like a system testing, whereby the whole application is tested as a black box. Unit test does not involve UI actions, whereas GUI testing involves simulation of keystrokes, mouse clicks or keyboard actions. And the most important of all, unit test is usually written by the same developers who write the logic code, whereas UI scripts are often written by other developers-testers. As such, the unit test test code is usually written in the same languages, with the same tools as the production code, whereas the UI scripts are more often coded using different IDEs using different languages. And that, contributes to the first problem of the UI tests.
Tools do play a huge part in programming, because a good tool can easily reduce the accidental difficulties of it and increase programmer's productivity. There are numerous automated UI tools available on the market ( such as WinRunner, TestComplete, QA Wizard etc), and a mature one should come with a scripting editor. For someone who is accostumed to Visual Studio or Eclipse, the scripting editor seems awfully inadequate. Living with an editor without refactoring, syntax highlighting and numerous other features is just painful.
The second problem with automated UI test is, it is slow to run-- compare to unit test. Unit test is fast, because you just have to load the relevant libraries and test them from your IDE (using tools such as TestDriven.Net). But for the UI test, you got to
- Compile the whole application
- Launch it
- Launch your Scripting IDE
- Attach the application to the IDE
- Start testing
A good IDE eases maintenance effort, since most programming is all about maintenance, a serious software development effort must always take it into account. But the poor state of scripting tool complicates the task:
- My scripting code is not properly formatted, the braces are not properly aligned and the if-else statement does not lie at the same column location, any shortcut keys to format the text? No.
- I need to debug my test scripts, but the debuggers are primitive and hard to use.
There are a lot of primitive actions that the developers-testers have to write. Actions such as clicking and dragging an objects are not wholely supported. Modern computer languages (such as C# and Java) abstract away these low level details so that the developers can concentrate on higher level things, but apparently the scripting technologies are not that advanced yet.
Capture/Replay Tool To Save the day?
You may ask, why all the trouble? Don't we have a fanciful capture/replay tool built into almost all of the automated test tools? There is no need to write scripts, you click a start recording button, do everything you want and press a stop recording button. Viola! You have a test case.
But these scripts are very fragile, and unmaintainable. The capture/replay tool is just like any other programming tool that is used to write program. It can never be as intelligent as human. Anyone using it to generate test scripts will have a hard time living with them.
And most of the time, the capture/replay tools cannot intelligently identify the tested objects. All they can do is to blindly record your action in terms of screen coordinates. Thus if your menus or icons change place, the tests break. In all fairness, some test tool vendors (such as TestComplete) can identify common objects to a certain degree by linking those objects with their names. But still, this capability is very limited.
Finally, there is a lack of interoperability and standards among different test tool vendors. This means that the scripts I write using Test Tool A cannot be run on Test Tool B. Although the tool vendors love to lock-in their customers for obvious short term reasons, but in fact it simply discourages people from practicing automated UI testing, resulting in a loss of potential customers and hindering the process made in removing the accidental difficulties of automated UI test.
None of the problems outline above are insolvable. All it takes to solve these problems is to have a more powerful scripting editor. There are other essential difficulties with automated UI tests. But that is a topic for another day.