Last week my brother and sister in law were visiting, so I mostly took the week off from writing. But I did ask you how reproducible your builds are.

I got several responses, of all flavors. Some were slightly appologetic, that their builds were mostly, or often, reproducible, but not always. Some didn’t try at all for reproducibility. One person said they build each release on three separate machines and operating systems, and do an MD5sum comparision before release!

But what I want to focus on here is the answers to the second part of the question I asked: Why or why not care about reproducible builds?

Not everyone who wrote to me answered this part, but I got enough to see some patterns.

The most common answer I got was that reproducible builds were desirable in case of rollbacks. The idea being that you may not be able to preserve all old artifacts, so you need to be able to reproduce them to revert to a known good state in some cases.

Another response was that it gives you confidence that the thing you tested is the thing you delivered to production. Although arguably, this may be achievable without reproducible builds, reproducible builds make it easier, as you don’t necissarily have to promote the same literal build artifact from testing to production.

Interestingly absent from most of the responses I got was security concerns. reproducible-builds.org touts security as the primary motivation for reproducbility:

The motivation behind the Reproducible Builds project is therefore to allow verification that no vulnerabilities or backdoors have been introduced during this compilation process.

I did have one response that said they do not care about reproducible builds, because they have other safeguards in place that mitigate this particular type of risk.

While I’m sympathetic to the security concerns, I find that most teams I’m helping, anyway, are struggling with much more pressing issues, and most security concerns are barely even a thought.

Having said that, there is still one reason I find reproducible builds to be compelling for many of the teams I’m helping. Although it can depend a lot on the complexity of what they’re building…

That reason is: Ease of debugging. Especially the system being deployed is complex, with many services, or complex dependencies.

There’s nothing worse than receiving a customer complaint about a bug, then go trudging through your logs to find the exact time and place the bug occurred, only to find you can’t reproduce it. So you check out the version of the code that the customer was running, and you still can’t reproduce it… Ugh!