Should we feel bad if we don't deploy on Friday afternoon?
Everytime we decide when to deploy, we're making a business decision. And few business decisions are black-or-white.“Friday deployments are utterly irresponsible!”
“You should be ashamed of your shoddy system if you can’t deploy on a Friday with confidence!”
Both of these extreme ignore a lot of nuance that should go into the decision of whether or not you deploy on a Friday (or any other given moment during the week). Yesterday I talked about some of the counter-risks to not deploying a security fix on a Friday afternoon. But that just barely hints at the nuance that we really should be considering.
Ultimately, everytime we decide when and if to deploy code, we’re making a business decision. And few business decisions are purely black-or-white.
It’s perfectly reasonable to have a highly reliable software system, with quick rollback options, and still choose not to deploy on a Friday afternoon. And dare I say, do so without shame! Likewise, you may be on a very unreliable system, and choose to do a Friday deployment anyway, for good reasons. Before I offer a hypothetical example of either, though, let’s talk about the different factors that can go into such deploy/don’t-deploy decision.
- Are there security implications of not deploying sooner?
- Are users waiting for the change? (i.e. a new feature, bug fix, or performance improvement)
- Are you or your colleagues waiting for the change? (i.e. adding instrumentation, or an experiment whose results you need to progress)
- Does your change touch code that others need to change soon? (i.e. waiting to merge will introduce merge conflicts)
- Is someone (ideally you!) able to respond (and possibly revert your change) if things go wrong?
- How confident are you that you will know if something goes wrong?
- What’s the worst-case scenario with this change?
- What’s your mitigation plan if things do go wrong? (roll back? Hopes and prayers?)
- How many other changes are bundled into your deployment? Can you adequately answer all of the above questions for each one?
I’m sure you could come up with a few other things to include in the list, but this should be enough to make the point.
Not every deployment is created equal. And while I’ve mostly been talking about “reliable systems” in my past messages on Friday deployments, that’s really only one variable (or more accurate: one set of variables). The change itself contains another set of variables. And only taken together can you make an informed decision about whether or not a Friday afternoon deployment makes sense.
With a highly reliable system at your finger tips, single-piece deployments, and quick rollbacks, a sufficiently trivial change may not be worth deploying on a Friday afternoon. Sure, the odds of failure are very low. But do you really want to stick around an extra 15 minutes just to be sure that your TODO comment typo fix deployed successfully, before you punch out and hit the pub? Maybe not. And I don’t think there’s any shame in that.
On the other hand, you may have a highly unreliable system, and you’re still doing big-batch deployments, but you absolutely NEED to get this deployment out for some reason. That can also be a legitimate business decision.