Software Testing Separation of Concerns

Introduction

Building quality software requires software testing. Granted, any testing no matter how off-base is better than zero testing. Without a firm grasp of certain testing concepts however, producing a correct and repeatable test suite becomes difficult and you can end up with a half-ass mix of somewhat useful tests. Just as developers (hopefully) strive to provide for the separation of concerns in application code via layering, object oriented programming/design, etc., so must you also be cognizant of and implement a clear-cut separation of concerns when it comes to your software testing strategy. To illustrate this, let’s look at what I call the Levels of Software Testing.

Level 0: Syntactical Testing

Although many developers would not think of the compiler or interpreter of their choice as a testing process and tool, these systems serve as the start of the testing journey: syntactical testing. If your code does not compile, build, link, or interpret then you have just immediately failed your test suite.

One could also argue that code that is poorly documented, structured, or just plain sloppy fails another test – the maintainability test. If the code cannot be easily maintained or refactored, then the code itself becomes a possible surface for the introduction of bugs.

On a side note, crashing your compiler also is an immediate test suite failure. Real world example: the code base on a recent consulting engagement written in VB.NET would consistently crash the VB.NET compiler due to its heavy use of generics. The code base was subsequently converted to C#, the compiler crashes became a non-issue, and overall quality increased by orders of magnitude Resharper helped immensely as well. Negative software quality manifests itself in mysterious ways.

Level 1: Unit Testing

Unit testing is a quite simple but very misunderstood testing process. MSDN, surprisingly, puts it very succinctly: “The primary goal of unit testing is to take the smallest piece of testable software in the application, isolate it from the remainder of the code, and determine whether it behaves exactly as you expect. Each unit is tested separately before integrating them into modules to test the interfaces between modules. Unit testing has proven its value in that a large percentage of defects are identified during its use. Finding the error (or errors) in the integrated module is much more complicated than first isolating the units, testing each, then integrating them and testing the whole.” [1]

In order to test the interfaces (i.e. point of interconnection) between modules, the use of interfaces (i.e. an abstraction of a software component), dependency inversion and injection principles, and mock and/or stub objects are the correct and preferred way to proceed. Leveraging these measures also has the desirable side effect of ensuring separation of concerns in application code as well. Note as well, if your code is not loosely coupled (and highly cohesive), then it will be difficult to do correct unit testing.

It is all too unfortunate that many developers still in this day just do not get what a unit test is designed to test and not test. Consider this Wikipedia excerpt that really drives it home: “Because some classes may have references to other classes, testing a class can frequently spill over into testing another class. A common example of this is classes that depend on a database [or other component]: in order to test the class, the tester often writes code that interacts with the database [or other component]. This is a mistake, because a unit test should never go outside of its own class boundary. As a result, the software developer abstracts an interface around the database connection [or other component], and then implements that interface with their own mock object [or using a dynamic mock library]. By abstracting this necessary attachment from the code (temporarily reducing the net effective coupling), the independent unit can be more thoroughly tested than may have been previously achieved. This results in a higher quality unit that is also more maintainable.” [2]

Examples of good unit test suites include those which exemplify:

• Are actually unit tests; if you see a test that connects to a live database or leverages an external component then that test is not a unit test; it is an integration test.
• Are automated and are able to execute headless (no one-off WinForms test harnesses) with code coverage enabled. This provides much needed introspection into how well the development team is adhering to the testing strategy.
• Should test the both positive and negative conditions. Failure to test edge cases and boundary conditions is a major cause of bugs which slip into QA and/or production.

Level 2: Integration Testing

Integration tests are really what many developers are incorrectly calling “unit tests”. The goal of integration testing is to expose defects in the interfaces and interaction between integrated units. Given a component A which depends on unit B and unit C, an integration test should be created to ensure A, B, and C play nicely in a live situation.

There are many ways to perform integration testing but generally the strategies come down to top-down, bottom-up, and functional-driven. A top-down approach entails starting your integration testing with integrated components as high up in the dependency space as feasible and working down until you have a full suite; a bottom-up approach is the opposite in that you start your integration testing with integrated components as low down in the dependency space as feasible and working up until you have a full suite. A functional-driven approach uses key feature code paths as the guiding light in integration testing.

Examples of good integration test suites include those which exemplify, as blogged by Jeffrey Palermo:

• “Smaller integration [versus large system tests] tests will help narrow the area where the failure lies.” [3]
• “An integration test must be isolated in setup and teardown. If it requires some data to be in a database, it must put it there.” [3]
• “It must also run fast. If it is slow, build time will suffer, and you will run fewer builds - leading to other problems. “ [3]
• “Integration tests should be order-independent. It should not matter the order you run them. They should all pass.” [3]

Level 3: System Testing

Relying on Wikipedia for a concise definition, system testing is “testing conducted on a complete, integrated system to evaluate the system's compliance with its specified [technical and non-technical] requirements. “ [4] During system testing, certain behaviors, artifacts, and/or aspects of the system are usually tested:

• Capacity, Scalability, Performance, Load, Volume, and Stress
• Security and Error Handling
• Installation, Maintenance and Recovery
• Accessibility, UI, Usability, and Help

System testing is can be done by developers, QA, or become a shared responsibility.

Level 4: Acceptance Testing

Acceptance testing involves performing automated or manual tests by the end-user, customer, QA team, or client to validate whether or not the product meets acceptance criteria. The end result is a decision to accept the product all, some, or none of the product as being feature complete and feature correct.

Conclusion

It is easy to see that the focus of testing starts at the smallest concern (source code) and continually broadens (units, components, system) to the largest concern (customer). It forms a pyramid like paradigm very well suited to structure a successful software testing strategy.

References:
[1] http://msdn2.microsoft.com/en-us/library/aa292197(VS.71).aspx
[2] http://en.wikipedia.org/wiki/Unit_testing
[3] http://codebetter.com/blogs/jeffrey.palermo/archive/2006/05/09/144387.aspx
[4] http://en.wikipedia.org/wiki/System_testing