Monthly Archives: April 2014

Testing in Perspective

With his keynote at RailsConf this week, David Heinemeier Hansson kicked the test-driven development (TDD) hornet’s nest again (video is linked at the bottom), re-igniting the always simmering debate over the role of testing in software development. While I don’t believe any of his arguments were particular new, he did talk about how he personally arrived at his current opinion on the role of testing and exposed his audience of Rubyists to the idea that the TDD gospel can be questioned.

I have also evolved to take a pragmatic approach to tests, nicely articulated by Kent Beck (DHH also cited this in his talk):

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don’t typically make a kind of mistake (like setting the wrong variables in a constructor), I don’t test for it.

– Kent Beck on stackoverflow

This quote is so popular because it asserts the proper perspective in a succinct and pragmatic way. The product is the project, not the tests. Use your judgement and write enough tests to serve the needs of your project. This pragmatic approach is shared by anyone who has ever been responsible for delivering an entire working product to a paying customer.

The “entire working product” part is important because this seems to be the part that unit-test enthusiasts miss as they focus almost exclusively on low-level interfaces. Most people I’ve talked to who have worked on big projects can tell you that bugs are found in core features by QA testers or customers on products with very high levels of test coverage.

Most projects have limited time and you need to get the most from the time you have.  If you’ve spent some of that time pursuing 100% unit test coverage, to the exclusion of system testing or other improvements to the product or code, it’s almost certainly not the best use of that time.

How Many Tests?

Zero is too few, and 100% code coverage (which still doesn’t cover everything) is too much. What’s just right?

The approach articulated in Beck’s comment is to optimize the value you get from the time you spend writing tests versus the time you spend writing code. So the question of “How Many Tests?” can be answered, “Enough for you to achieve the benefits of automated testing,” many of which are about saving time on the larger project. To name just a few:

  • Automated tests give developers confidence in adding new features or refactoring existing code because running the test suite gives an initial baseline that core functionality hasn’t been compromised.
  • Tests can help define a feature or interface, helping the developer verify they have provided all requested features, especially on a large team.
  • Tests written to exercise specific bugs protect against regression, at least for those specific issues.

All of these are benefits because they ultimately save time. However, you begin to lose some of this time savings if you spend too much time pursuing 100% test coverage.

Designing Testable Code

Another point DHH makes in his keynote is that he feels that making design decisions solely to serve the purpose of testing can lead to clunky or poor code design. Again, my initial thought was that this wasn’t particular controversial and he was just stating one of many factors that any working programmer already considers every day. But TDD does speak to the role of testing in code design, so I can see where some developers would take issue with designing code that might be hard to test.

In addition to providing the benefits noted above, one of the original promises of TDD is that it would actually lead to more modular, and therefore better designed, code. Like all other best practices, this isn’t universally true. Programmers have to think about a hundred different factors when designing code, all while dealing with the fundamental challenge of creating the new features needed for their product and making them work right. Making the code testable is just one of these factors, and it does come naturally if you are testing as you go along.

As an example, brian d foy has popularized the concept of a “modulino” in his Mastering Perl book, in code, and in various talks and presentations. This design pattern makes command-line scripts written in Perl much more testable by encapsulating the code in methods you can call from test files. When writing a command-line script, you need consider whether the additional code, and perhaps slight bit of obfuscation, outweighs the benefits of easier tests.

  • Is the script complicated with lots of options? Set it up as a modulino so you you can write some tests for the hard bits.
  • Will it be around for a long time with many users? Use a modulino to make it easy to add tests in the future as it expands.
  • Is it a very simple script with limited functionality? Maybe skip the extra modulino setup.

The point is that testability is one of many factors and you need to assess and weigh all of these when you’re writing code, including how it impacts the design for you and other future maintainers. More experienced programmers will be aware of more factors and will do a better job assessing the importance of these factors over the life of the code.

Testing against components external to your program, like databases, web services, etc., can lead to some tough decisions in terms of how much to bend your design for testing. It also can lead down the road of mocking parts of your system and how significantly you want to compromise your tests as you start to mock the environment.

When you write software, the goal is to create working, performant software that can be maintained, extended, and expanded over time. You need to do so within the time and budget that makes sense for the end users. When thinking about the patterns and support frameworks you’ll use, including testing, you need to keep this perspective. You need to do some level of cost-benefit assessment to decide how much effort to put into these support structures, and the more experienced the programmer, the more accurate this analysis is likely to be. While there is no question that automated testing should always be part of this analysis, pursuing TDD as a goal in itself can lead to costs out of proportion to the benefits and run counter the overall goal of creating a useful, compelling product for users.