Test-Driven CI pipelines

Test-Driven doesn't only apply to normal code.

Whenever I add a check to my CI pipeline, I try to follow “Test-Driven” practice. That is, I make my CI pipeline fail, before I fix whatever thing made me want to add the new check.

This happened to me recently when I recently discovered a broken page on my web site. I had a typo in the template language, which left some of the templating code un-rendered. I had intended to write something like:

%%< img src="foo.jpg" >%%

But instead had:

%< img src="foo.jpg" >%%

Which rendered literally on the page, rather than showing a pretty image of foo.

And as I am known to do, I wanted to solve this problem twice, once by fixing the typo, and a second time by preventing such typos from occurring again. So I added a little check to my CI script:

  - find ./public -type f -name "*.html" -print0 | xargs -0 grep -n '%%' && exit 1

It just makes sure my generated HTML output doesn’t contain %%. Not perfect, but an improvement.

The only problem?

It didn’t work.

It turns out a bug in the grep version I was using in CI didn’t set the right exit status when a match was found. So although it was finding a violation in the output, the CI pipeline didn’t exit with a failure. The solution was to use a different base docker image that didn’t have that bug.

If I had just assumed that local testing was sufficient, I would have added a completely useless check to my CI pipeline.

Share this