Well, some good came from it
After a failed monster deployment, we're finally switching to continuous deployment.Last week I related the story of a monster release gone bad. 167 changes, and we didn’t know which one was to blame.
Well, the next chapter in that story is a pleasant one.
We decided to re-prioritize the task of migrating to a modern infrastructure, so that we can begin doing continuous deployments.
You see, the current infrastructure is a bit of a black box. It was put in place by a team that hasn’t been there for years. Nobody working there understands the current system well enough to manage it or change it. And while I’m never one to advocate a re-write or re-platforming effort out of ignorance of the current system, key parts of the old deployment process depend on binaries for which the source code has been lost, making it impossible to safely update or replace those portions of the process.