What to do with broken tests
What should you do with broken tests on an old project? Delete them. Probably.Imagine you’ve just started a new job or joined a new team. They have a working product, that’s already used in production by a large number of active users, but the code looks like a big pot of grandma’s spaghetti, complete with tomato sauce and meatballs.
This describes the majority of projects I’ve joined, so I expect you’ve run into something similar before, too.
Now further imagine that this project has a bunch of automated tests written, but they aren’t run regularly. Perhaps because they take too long (hours?), or perhaps simply because nobody took the time to set up a build server and CI pipeline to run the tests automatically.
Whatever the reason for the test disarray, how should you proceed, if your goal is to start receiving value from these tests again?
Below is my suggested approach, which I have used successfully on several projects:
-
Disable broken tests.
The first step is to get your tests into a passing state. Do this quick and dirty. DO NOT try to fix any broken tests at this stage. That comes later. Only disable, skip, or comment-out any broken or flaky tests. This should take no more than an hour or so, regardless of the size of your code base. If it takes longer than this, you’re spending too much time dwelling on details. Start disabling all the tests for entire modules, classes, or other large units of code. The goal here is not to have as many working tests as possible, but rather to have no broken tests running.
-
Automate your tests.
Now that you have a set of known good tests, get them running automatically for every build or every pull request. From this point on, every failing test should be fixed before a new release.
-
Fix broken tests.
Now is the time to go back through all the tests you disabled, and see which ones can be fixed without a lot of effort. Don’t spend too much time on this, though. Depending on the code base, sometimes up to half of the broken tests can be easily fixed with some minor tweaks. As a very rough rule of thumb, don’t spend more than 5-10 minutes trying to fix any individual test.
-
Delete broken tests.
If you still have broken tests after a minimal effort at fixing them (and you probably will), just delete them. If you’re nervous, leave the disabled tests in your code base for up to a month or two, maxiumum, then delete them.
Feeling nervous about it? Remember:
- Nothing is ever truly deleted from version control (you are using version control, right?)
- Your code is already tested, by virtue of the fact that it’s running in production. The automated tests are just a bonus at this stage, so deleting them doesn’t really cost you much.
-
Write tests for all functionality changes.
Now this is where the actual magic happens. Every time functionality changes in the code, add a new test. Fixing a bug? Add a test to ensure it doesn’t regress. Adding a new feature? Write tests for the new feature.