Blog / Software Delivery

Roll with the changes

In my dreams, every new feature I develop is amazing. Customers love it, and they receive great value from its use.

Yesterday I explained why rolling back is dangerous in these situations. Today I present you with my preferred alternative:

Rolling forward

The idea is fairly simple. When a critical failure occurs, rather than rolling back to a previously-built version of the software, we revert the broken change in git (or other VCS), then build and deploy a new version of the software.

In such a case, our scenario looks something like:

  1. Service version 1.2.4 is released to production. A serious bug is discovered.
  2. The broken commit or pull request is immediately reverted in git, then a new version, 1.2.5 (which is likely functionally identical to 1.2.3) is built and deployed to production.
  3. Ongoing work by other developers or teams continues as usual, unhindered, and versions 1.2.6, 1.2.7, etc, are released.
  4. Eventually the bug fix is applied, and version 1.2.8 (or 1.3.79 or whatever) is released.

With this approach, the project continues to enjoy the benefits of continous deployment even while solving a production-critical bug. Of course, this assumes that the team is using real continuous integration, and continuous deployment—if your team isn’t, you should start considering that change first!

If this is something you and your team could benefit from, you’ll find my video course Ship better code faster valuable. Use coupon code DAILYMARCH to save 50% through the end of March.

Share this

Related Content

The two big dangers of rolling back

Broken code is inevitable. Despite our best planning, automated tests, code review, and everything else, sooner or later (probably sooner) you’ll find that you’ve deployed a broken feature into production, and users of the system are complaining. Our natural instinct in such a case is to quickly jump back to the last working version—a process we colloquially call a “rollback”. Today I offer two reasons not to do this. Tomorrow I’ll talk about an alternative.

Well, some good came from it

After a failed monster deployment, we're finally switching to continuous deployment.

Think unnaturally

We humans take a lot of natural ideas with us from the material world, to the software world where they aren't context appropriate.