Test-Driven CI pipelines

June 6, 2022
Test-Driven doesn't only apply to normal code.

Whenever I add a check to my CI pipeline, I try to follow “Test-Driven” practice. That is, I make my CI pipeline fail, before I fix whatever thing made me want to add the new check.

This happened to me recently when I recently discovered a broken page on my web site. I had a typo in the template language, which left some of the templating code un-rendered. I had intended to write something like:

%%< img src="foo.jpg" >%%

But instead had:

%< img src="foo.jpg" >%%

Which rendered literally on the page, rather than showing a pretty image of foo.

And as I am known to do, I wanted to solve this problem twice, once by fixing the typo, and a second time by preventing such typos from occurring again. So I added a little check to my CI script:

  - find ./public -type f -name "*.html" -print0 | xargs -0 grep -n '%%' && exit 1

It just makes sure my generated HTML output doesn’t contain %%. Not perfect, but an improvement.

The only problem?

It didn’t work.

It turns out a bug in the grep version I was using in CI didn’t set the right exit status when a match was found. So although it was finding a violation in the output, the CI pipeline didn’t exit with a failure. The solution was to use a different base docker image that didn’t have that bug.

If I had just assumed that local testing was sufficient, I would have added a completely useless check to my CI pipeline.

Related Content

An unexpected benefit of short feedback loops

Practices that create shorter feedback loops help gamify the software creation process, and more frequent dopamine releases.

Talk Notes: Scaling CD Down

The poor man's feature flag

How to use feature flags if you don't have feature flags.