Search This Blog


Thursday, December 27, 2007

Accidental difficulties with Automated UI Test

Automated GUI Testing is always a pain in neck. There are many things that one can talk against it. But this post will concern itself with the accidental difficulties of the automated UI test.

Before I start my rant I would like to make a difference between Automated GUI Testing and unit testing. There is a lot of differences between the two. Unit test tests against a component; its scope is often limited to class libraries. But GUI testing is more like a system testing, whereby the whole application is tested as a black box. Unit test does not involve UI actions, whereas GUI testing involves simulation of keystrokes, mouse clicks or keyboard actions. And the most important of all, unit test is usually written by the same developers who write the logic code, whereas UI scripts are often written by other developers-testers. As such, the unit test test code is usually written in the same languages, with the same tools as the production code, whereas the UI scripts are more often coded using different IDEs using different languages. And that, contributes to the first problem of the UI tests.

Tools do play a huge part in programming, because a good tool can easily reduce the accidental difficulties of it and increase programmer's productivity. There are numerous automated UI tools available on the market ( such as WinRunner, TestComplete, QA Wizard etc), and a mature one should come with a scripting editor. For someone who is accostumed to Visual Studio or Eclipse, the scripting editor seems awfully inadequate. Living with an editor without refactoring, syntax highlighting and numerous other features is just painful.

The second problem with automated UI test is, it is slow to run-- compare to unit test. Unit test is fast, because you just have to load the relevant libraries and test them from your IDE (using tools such as TestDriven.Net). But for the UI test, you got to
  1. Compile the whole application
  2. Launch it
  3. Launch your Scripting IDE
  4. Attach the application to the IDE
  5. Start testing
So many steps! With unit test we can talk about letting the test drives the development, but we can't do that with UI test. Every time we make changes to our code, we can run all the unit test cases before we check-in. But due to the slowness nature of UI test, we can only run it after we have a daily build.

A good IDE eases maintenance effort, since most programming is all about maintenance, a serious software development effort must always take it into account. But the poor state of scripting tool complicates the task:
  1. My scripting code is not properly formatted, the braces are not properly aligned and the if-else statement does not lie at the same column location, any shortcut keys to format the text? No.
  2. I need to debug my test scripts, but the debuggers are primitive and hard to use.
The list just goes on and on. Your automated test tools may be different from mine but I am sure that they are not as advanced as the standard programming tools. And that's a serious problem because it makes writing automated test scripts expensive.

There are a lot of primitive actions that the developers-testers have to write. Actions such as clicking and dragging an objects are not wholely supported. Modern computer languages (such as C# and Java) abstract away these low level details so that the developers can concentrate on higher level things, but apparently the scripting technologies are not that advanced yet.

Capture/Replay Tool To Save the day?
You may ask, why all the trouble? Don't we have a fanciful capture/replay tool built into almost all of the automated test tools? There is no need to write scripts, you click a start recording button, do everything you want and press a stop recording button. Viola! You have a test case.

But these scripts are very fragile, and unmaintainable. The capture/replay tool is just like any other programming tool that is used to write program. It can never be as intelligent as human. Anyone using it to generate test scripts will have a hard time living with them.

And most of the time, the capture/replay tools cannot intelligently identify the tested objects. All they can do is to blindly record your action in terms of screen coordinates. Thus if your menus or icons change place, the tests break. In all fairness, some test tool vendors (such as TestComplete) can identify common objects to a certain degree by linking those objects with their names. But still, this capability is very limited.

Finally, there is a lack of interoperability and standards among different test tool vendors. This means that the scripts I write using Test Tool A cannot be run on Test Tool B. Although the tool vendors love to lock-in their customers for obvious short term reasons, but in fact it simply discourages people from practicing automated UI testing, resulting in a loss of potential customers and hindering the process made in removing the accidental difficulties of automated UI test.

None of the problems outline above are insolvable. All it takes to solve these problems is to have a more powerful scripting editor. There are other essential difficulties with automated UI tests. But that is a topic for another day.


Cory von Wallenstein said...

Agreed. There are many, many difficulties when doing automated testing of graphical interfaces. For us, it was becoming more of a burden than a boon to productivity.

It got much, much better when we somewhat formalized what should be tested with what strategies. The overall outcome was to only test the most likely to break in the graphical interface. When our test scope was too great, they became unmaintainable, as you aptly pointed out. We really concentrated on the low-hanging fruit for graphical testing. All of the actual workflow testing was accomplished with unit testing; the graphical tests are more "does this button do what its supposed to", and nothing more.

Life got better since then!

Soon Hui said...

Cory, I can't agree more with your point.

Indeed, we only use Automated UI test when we have to, when we need to test the interaction of GUIs.

We use unit tests to test other parts of the software components.

Anonymous said...


Soon Hui said...

Anonymous, I don't get what you mean.

Are you recommending me to use WATIR for UI test? Thank you for your suggestion, but I am doing desktop application. I afraid a web base testing framework won't work.

Anyway, thanks for your suggestion.