Simpler Tests: What kind of test are you writing?

One common problem that I see in test suites is confusion about what each test should cover. This confusion often leads to tests that are either too broad or too focused to accomplish the goal of the author. When writing a test, it is important to think about what kind of test it will be and the constraints that make that type of test effective.

There are a few broad categories of tests that I keep in mind to help focus my testing:

Unit Tests

Also called micro tests, unit tests form the foundation of any good testing strategy. These are tests that test a single class or method in isolation from it's dependencies. Mocks and stubs help us isolate the code under test and make our tests more focused and expressive. A test that touches any external resource such as the database, the file system, a web service, etc is never a unit test.
The primary purpose of unit tests is to validate that the subject under test is functionally correct for the expected inputs and outputs. Additionally, unit tests provide documentation on how the class is expected to function and be used by consumers.
Good unit tests are fast, atomic, isolated, conclusive, order independent and executed automatically by the continuous integration server on each commit to the source control system.
Smells for unit tests include too much setup, long test methods, race conditions, reliance on strict mocks, lack of CI integration.

Integration Tests

These are tests that include a specific concrete dependency. There are two primary sub-categories of integration tests. The first category of integration tests is the tests on your adapters for the features of another system. Examples include tests of your data access layer, web service proxies, file system proxies, etc. In order to provide real value, these tests must use the real dependency. For that reason, they often require more set up and are slower to write and run than unit tests, but in order to have confidence in your test suite, these tests are absolutely necessary.
The primary purpose of these integration tests is to validate the code that you have written to manipulate the external system. Additionally, these tests may validate some portion of the remote system. As an example, consider a data access object that is implemented in terms of an object relational mapper with stored procedures in the database. If you call Save, then Load and validate that the retrieved object is equivalent to the stored object, you have tested your DAO, your OR mapping, the stored procedure(s), the network connection, the database connection string, the database engine, the script evaluation sub-system in the database, etc. You can see that these tests exercise a larger block of code and infrastructure than unit tests. For this reason, they tend to be more brittle and prone to breaking. I don't recommend failing the build on failed integration tests. Once your suite of integration tests grows large, you may not even want to run them on check in, but just schedule them to all run at various intervals (hourly/daily.)
Good integration tests validate the features of the external system that you use in your application. They do not attempt to cover the full set of functionality, but only to validate that what you need works. Like your unit tests, these tests should be atomic, isolated, conclusive, order independent and executed automatically by the continuous integration server as often as is reasonable. You also want them to be as fast as possible, but keep in mind that they will never be as fast as your unit tests.
Smells for these tests include too much setup, long test methods, race conditions, reliance on any mocks or stubs, lack of continuous integration.
The second category of integration tests are those that depend on more than one of your own components. These are tests that are generally confused about their role. JB Rainsberger gave an excellent talk about these tests that you can watch here on InfoQ: Integration Tests are a Scam.

Acceptance Tests

Acceptance tests are often overlooked, but critical to a solid test suite. These are tests that execute your entire stack; excluding only the user interface. To be clear, this means using no mocks or stubs or test doubles of any kind. Your tests should attach at the Application Layer just under the UI Layer. These tests complement the unit and integration tests. While unit and integration tests validate correctness in the small, these tests validate correct composition of your building blocks. Acceptance tests are often written using different tools than those used in unit and integration testing. This is because in a mature environment, the specification for the tests can be understood, discussed and even written by users, business analysts, and QA testers.
The primary purpose of these tests is validate things like component wire up, application stack integration, basic use cases/user stories, system performance, and overall application stability. These tests will likely be run the least often as they will be fairly time consuming to execute and may require extensive (though automated) setup.
Good acceptance tests can be understood by a user and are written in terms common to the business.
Smells for acceptance tests include attempting to validate every path through the system and writing them in a programming language not understood by the users.

User Interface Tests

These are tests that manipulate your application the way the user would. In theory, you can test just the UI this way, but in practice, this most often means that these tests exercise the whole application stack. Writing UI tests often requires special tools to manipulate the application. These are the most fragile, brittle, expensive tests to write and maintain. They are also the slowest to run. It is often best to use these tests only to validate that the application is navigable, doesn't fall over and has all of its dependencies met.
Good UI tests are simple and limited in scope.
Smells for UI tests are inconsistent failures, testing application correctness.

TATFT

There are many ways to categorize tests. I find these 4 categories help me to focus my testing and build a faster and more reliable test suite. What techniques do you use to improve your testing suite?

** There are many tools that can be used to help build a fast, effective test suite. I intentionally avoided any mention of them by name in this post as a thorough treatment of the tooling would make the post many times longer. **