Search This Blog


Tuesday, November 11, 2008

Different Types of Automated Tests

When I first learnt about NUnit , I thought that all the tests that are written in NUnit are unit tests. And all the tests that are written using other tools and languages-- including scripting tools are integration tests. Now I know how wrong I was.

NUnit was designed to run unit tests. But due to its simplicity and versatility, it can be used for other automated testing purpose as well. So, in this post I am going to list down the kind of tests that can be run by NUnit or other similar testing tools.

Unit Tests
Without a doubt, unit tests are the main reasons why JUnit and NUnit existed. Unit tests are small, atomic; they are supposed to test the behavior of methods, not the whole system. Ideally there should be at least one test correspond to one method. Unit tests should be the first thing to run when one finishes the coding. In order words, before one puts link the Windows Form to the code, or before one integrates the business logic into the web application and see the results in a web browser, the corresponding tests should already pass.
Unit tests should heavily make use of mocks to stub out the irrelevant portion so that every part of the code can be tested thoroughly. Unit tests shouldn't concern themselves with the behaviors of external dependencies. When I first started on unit tests I didn't know the difference and just included all the external dependencies anyway. And when I got failures I usually had to track down the culprits and get them to fix their problem! Although this strategy was catching a significant number of bugs, but now I think that I was not doing the unit testing in the proper way. I was doing integration tests.

Integration tests
Integration tests are the tests that test the interaction between different components. Unlike unit tests, they take a longer time to run, and cannot be used in TDD. Integration tests are testing whether your application can stand up as a whole. It is much more comprehensive than unit testing and should catch more regression bugs than the unit tests.
Looking back, I think that the majority of tests I wrote for other people are integration tests, not unit tests. One can of course, use scripting tools to write integration tests. But fundamentally writing integration tests in Nunit or JUnit provides some benefits that other scripting tools don't:

  1. You get to develop your tests in your favorite language, not other third party propriety scripting language that may one day be discontinued.
  2. You get all the benefits of your IDE, which means that you can have equal access to the IDE features such as refactoring, code analyzing etc.
Verification Tests
Verification tests are the proofs to show that your code is actually conforming to the specification. Yes, you must have automated tests to show that the features you are boasting are actually working. Software features that can't be demonstrated by automated tests simply don't exist . Having relying on manual tests to convince myself that certain features were still working after code changes, I understand, first hand, how unreliable manual tests are. I am not saying that testers are not good. Rather, what I am trying to emphasize is that testers' time is limited; they can't possibly cover every single minor details when conducting software verification. Also, testers can easily suffer from fatigue and miss the obvious points if you ask them to test the same thing over and over again. Testers have their blind spots as well. You can't ask them to repeat all the verification steps for every daily build you make or else they will simply leave.

Computers, on the other hand, won't grow tired. They won't grow fatigue, they won't leave ( OK, they will break down, sometimes, but you have source control right?). So make use of that. Write automated tests one time only and let the computers run them every day.

And where to write the tests, except in NUnit or JUnit?

So, here are the three types of automated tests. Now, what are you waiting for? Go and write your code-- and the tests!

No comments: