Test-Driven CI pipelines
June 6, 2022Test-Driven doesn't only apply to normal code.
Whenever I add a check to my CI pipeline, I try to follow “Test-Driven” practice. That is, I make my CI pipeline fail, before I fix whatever thing made me want to add the new check.
This happened to me recently when I recently discovered a broken page on my web site. I had a typo in the template language, which left some of the templating code un-rendered. I had intended to write something like:
%%< img src="foo.jpg" >%%
But instead had:
%< img src="foo.jpg" >%%
Which rendered literally on the page, rather than showing a pretty image of foo.
And as I am known to do, I wanted to solve this problem twice, once by fixing the typo, and a second time by preventing such typos from occurring again. So I added a little check to my CI script:
- find ./public -type f -name "*.html" -print0 | xargs -0 grep -n '%%' && exit 1
It just makes sure my generated HTML output doesn’t contain
%%. Not perfect, but an improvement.
The only problem?
It didn’t work.
It turns out a bug in the
grep version I was using in CI didn’t set the right exit status when a match was found. So although it was finding a violation in the output, the CI pipeline didn’t exit with a failure. The solution was to use a different base docker image that didn’t have that bug.
If I had just assumed that local testing was sufficient, I would have added a completely useless check to my CI pipeline.