Code coverage percentage is overrated
“Percent code coverage” seems to get a lot of attention. A common question, anywhere programmers congregate, seems to be “What’s the ideal test coverage percentage?”
Some say 80%. Some “purists” say 100%.
What’s the right answer?
Well, what do these numbers represent? Typically (depending on the exact language/tooling you’re using) it represents the percentage of lines, conditional branches, or statements executed during the execution of a test suite. In theory, proponents say, 100% test coverage would mean that you’ve tested 100% of possible execution paths.
However, reality falls short of this theory. Here are several reasons why:
- It’s easy to execute code in a test, that isn’t actually tested. This is in fact easy to do by accident.
- In some languages, there are often bits of code that are entirely unreachable. This means you may see a sub-100% report that actually reflects 100% of reachable code.
- Some will argue this is a stretch, but on a modern computer and operating system, no program executes in a completely deterministic way, meaning that no percentage of test coverage can account for all possible execution scenarios. Even the simplest “Hello world”, without any conditional logic, depends on the OS scheduler, and must wait for I/O before displaying its greeting. As such, on a multitasking OS, no two runs of “Hello world” will behave exactly the same (in particular with regard to execution time).
These caveats aside, even if you were able to achive 100% coverage, what value would it provide?
Would this number ensure that our code is ready to ship?
Would it help us find broken code if our tests ever fail?
In a word: No. 100% test coverage, by itself, isn’t valuable. It’s overrated.