Are friday deployments "risky"?
It's only risky if your system is unreliable.Last week I talked about a recent Friday afternoon deployment, to fix a security issue.
A few people took issue with this, making sweeping statements like “that’s risky!”
So lets talk about that. Let’s talk about risk.
What is it about deploying on a Friday afternoon that might be seen as “risky”? I don’t know what every commenter had in mind, but I can make some guesses based on past conversations. I think there are generally three categories of risk people are talking about:
- The deployment may fail, or may include a bug. This risk isn’t unique to a Friday afternoon deployment, but may be compounded by the other risks.
- We may not detect the failure or bug quickly or reliably, and it may go unnoticed (possibly all weekend)
- The people with the skills, knowledge, and access credentials to resolve the problem may not be available on the weekend.
Now if you work on a system where any one of these risks is high enough, you would be well justified in not wanting to release on Friday afternoon. But those aren’t the only risks at play. Particularly in the security-patch situation I described last week. In that case, we had a couple of other risks to consider:
- What if someone exploits this newly discovered security problem over the weekend? Admitedly, this had a very low probability, considering the vulnerability had apparently existed for months or years without having been exploited, but potentially high impact.
- What if clients, or regulators, learn that we knew of the security vulnerability, but sat on our proven fix, without deploying it, just because it was Friday? Also low probability, but could have a potentially high impact as well.
Making blanket statements like “It’s too risky to deploy on a Friday afternoon” is just absurd.
It’s only risky if your system is unreliable. The monitoring, alerting, rollback procedures, and people involved, etc, are all part of the system.
True, for some systems, the risks of deploying an urgent security fix on a Friday afternoon may well be too high to be worth taking. But if you’re not working on mitigating those risks, what are you doing?