Why I test, and why do you test?
This is most of an email I sent to an internal Cogent mailing list after an entertaining exchange between a couple of the guys about testing styles/philosophies. Craig suggested I post it here, so here ’tis :)
– snip –
First about unit and integration tests.
- I write unit tests for focused feedback; i.e. tell me exactly what broke. To keep them focused I try to keep them orthogonal which usually means using fakes of any collaborators.
- I write integration tests where I need more safety than a unit test will offer. They seem even more important when I’m stubbing and mocking a lot in a dynamically typed language like Ruby.
- I write both types of tests to help convey intent and understand the problem better. I TDD with either a unit test or integration test, whichever feels natural.
Then about interaction and state based testing.
- I pick the approach that feels natural at the time, favouring neither by default. I struggle with rules about when to use which.
- I dislike interaction tests that look suspiciously similar/symmetrical/coupled to the code they refer to. I expect a test to earn its right to exist, and therefore add to the size of the codebase and build’s time, by either conveying intent that is difficult to express in the code itself (which is why I love the term ‘example‘) or addressing some other consciously identified risk.
Your thoughts?
Ive been feeling uncertain about the merits of doing lots of automated testing of late.
At IBS now, we are approaching 100% test coverage through fairly orthodox TDD. What Ive noticed is that (a) even using latest- and greatest- techniques, we are spending lots of time writing mind-numbingly boring tests to test-drive simple code, (b) the tests are basically regression tests that assert the code does what it does, (c) they obstruct us when we want to change stuff.
Honestly, if you just want to verify a piece of code works, it can be cheaper to use a big-bang integration test, or watch it execute in a debugger, than to write unit tests over every branch.
So it seems that tests are really most valuable in preventing regression. Thus one’s “payback” from faithful TDD is perhaps accrued slowly as the code is used and modified. If TDD code is changed or discarded soon after its written, it seems that economically, TDD wasn’t then worth it.
benhutchison
2 Apr 08 at 9:54 pm
Any ceremony can start to seem pointless without revisiting its purpose from time to time.
A big part of why I write tests is initially to clarify my own understanding and later to communicate intent to anybody who might want to change this code.
These tests as a regression suite relates to the latter only, so I find my tests can look one way when I’m exploring at first and quite different when I’ve finished. I often find the first few tests look more like functional tests, but the ones I leave behind are a larger group of much more focused (unit?) tests with only a few functional tests.
Systematically producing long running tests is going to slow down the build considerably and, as I’m sure you’ve experienced, this can be debilitating in an agile team - but that’s another discussion :)
hiremaga
12 Apr 08 at 9:10 pm
I know what you mean about mind-numbingly boring tests. Still, I think they give more safety than the other options you mentioned. Watching code execute in a debugger might be fine to verify it the first time, but who does that repeatedly every time a change is made that might affect that stretch of code? Tests need to be automated.
Big bang integration tests might make a good substitute, IF they actually execute all the code paths. I haven’t done much with these, but they seem like they would take even longer to run than a big suite of unit tests… especially if I just want to test the change I made to one class, and have to run the whole integration test suite to do so.
Having regression testing can help in a couple different scenarios. One is when you need to make a change to a piece of code that some other developer wrote; it can be hard to know for sure if your change will break something, but a good automated test suite may help you catch things like that sooner. The other is refactoring. If you have a good test suite, you don’t need to be afraid to refactor your code, whether for performance, clarity or to get to a better design as the codebase evolves.
Test Driven Development also has the design benefits from having written the tests before the code, but I won’t go into those right now.
Wes Sheldahl
17 Jul 08 at 2:24 am