This episode is a replay of my Scrum Day Europe 2021 presentation, Scrum Isn't Enough: Why DevOps is essential for Agile success.

Resources:
Scrum Day Europe 2021
Presentation slides
Book: Agile Software Development with Scrum
The 2021 Scrum Guide
Book: The Phoenix Project
Book: The Unicorn Project
Free email Lean CD Bootcamp

Transcript

Announcer: Ladies and gentlemen, the Tiny DevOps guy.

[music]

Johnathan Hall: Hello and welcome to another episode of the Tiny DevOps podcast, where we believe you don't need a thousand engineers to do world-class DevOps. I'm your host, Jonathan Hall, and today, rather than having a guest on, I decided I would share with you the presentation I gave several weeks ago at the Scrum Day Europe conference entitled Scrum Isn't Enough (Why DevOps is Essential for Agile Success).

Of course, when I gave this presentation at the conference, there was a slide deck. I will have a link to those slides in the description, or you can hop over to my YouTube channel and watch this presentation with the slides included. Now, when I first gave this presentation, actually, about two or maybe even three years ago at a small conference, one of the other conference presenters, I overheard him saying during one of the breaks before I spoke, I heard him say, "Scrum is DevOps."

Now, I don't know who this guy was. I don't remember his name, but one of the things I'd really like to convince you of today is that that's not true. That Scrum is very distinct from DevOps, but that's okay and that's good. I'm also going to try to convince you that Scrum is incomplete and that that's also okay, and that DevOps helps to fill an essential gap in this Scrum story. At the end, I'm going to give you a really brief way you can start with DevOps if you're new or your team is new to the concept or to the practices. Just one quick suggestion for maybe how you can get started in that way.

Let's start with explaining the title. Why Scrum isn't enough? One of the very first books on the topic of Scrum was called Agile Software Development with Scrum. It was released within just a few months of the signing of the Agile Manifesto written by Ken Schwabe and Mike Beedle. If you go to amazon.com today and look for this book, the book is out of print, but you can still buy used copies.

The sales text for that book says, and I'll just read a quote from it, "eXtreme Programming is an ideal many software shops would love to reach, but they cannot actually implement it. This book shows readers how to use Scrum to implement XP in their shop." This is interesting to me because from the very outset, Scrum was apparently viewed as an incomplete system for software development. It was viewed as a way to enable other things, other technologies, other processes.

Now we'll jump to more modern times, the 2020 Scrum Guide says, "The Scrum framework is purposefully incomplete." Not only was Scrum incomplete at the beginning, it is still considered by the designers to be incomplete. What are some things that Scrum doesn't address? Obviously, one of those things that's really relevant to a DevOps audience is that Scrum doesn't tell us how to deploy our software.

It doesn't tell us if we should deploy to Kubernetes or if we should deploy to floppy diskettes or to C-ROMs or if we should upload to an FTP server, use Docker, AMI images, it tells us nothing about any of these. The closest it comes to talking about how to deploy software is that we should have working software after each sprint, which is typically two weeks. The closest that Scrum comes to telling us about deploying software is that we should have something every, roughly, two weeks.

Scrum also doesn't tell us how to handle customer feature requests. Should customers send feature requests straight to Jira through some web handle or web hook? Should they go through a customer service department? Should they call the CEO at home while he's having breakfast and asked for a request that way? Scrum doesn't have any opinion on this. Scrum doesn't tell us which programming language to use. If we're using C# or Java or Python or Ruby, Scrum simply doesn't care.

Obviously, Scrum doesn't tell us what spoken language to use. You can use Scrum if you speak English, if you speak Spanish, if you speak any other language or a mixture of languages or it changes from day-to-day depending on the colleagues you're with and so on. Obviously, something relevant these times is that Scrum doesn't have any opinion on where we should sit physically. Should we be in the same office, in the same city, in the same country or time zone? Should we all be working from home? Opinions are bound on this topic, but Scrum doesn't talk about this.

Scrum should work or should be made to work in any of these various situations. Perhaps most disappointingly, Scrum doesn't tell us the best color of sticky note. We all know that blue is the best color, but we don't learn that from the Scrum. We had to learn that objectively through other means, of course. Of course, there's a million, million other things that Scrum doesn't talk about. That's just a small taste of them.

When I'm talking about DevOps to Scrum practitioners and introducing a concept for the first time, I like to talk about how Scrum and DevOps fit in an organizational chart. Not because it's necessarily the best model, but because it's an easy model to understand. I usually draw, on a whiteboard or chalkboard, something that looks like a typical organizational model, sort of a pyramid shape with maybe a CEO or an executive suite on top, some sort of middle management layer, and then at the very bottom, of course, we have our individual contributors that can be developers, UI designers, QA testers, copywriters. It depends a lot on what your business is doing, but all these individual contributors are at the bottom.

I usually say that Scrum manages the lower or middle management layer the way they interact with the people on the bottom. I know this is a gross oversimplification and some of you Scrum elites out there are yelling that that's not the way this works. Bear with me. This isn't meant to be a complete or a hundred percent accurate picture. It's meant to help us grasp the concepts of how Scrum and DevOps interrelated.

If we're willing to accept for the moment that Scrum manages the relationship between lower or even product management and the individual contributors, then what's left. One of the obvious things that might be left is how do you handle upper management to middle management? There's all these conversations we could have about scaling agile. I'm not going to talk about that today.

Instead, I want to extend that mental picture we have of the organizational chart down one layer lower than the individual contributors. What's there? In my model, what's at that lower level are our technologies, are in our computers and our servers. Things like the language of choice, whether we're deploying to Amazon or Azure, Docker, Kubernetes. You can have technical practices in there such as code review and peer review and test-driven development, your backup plans, disaster recovery, on-call policy. There's a thousand things you could put down there. These could be thought of largely as things that individual contributors do or the things they use.

The way I like to look at DevOps, is it DevOps-- and in a sense also practices like eXtreme Programming, XP, address this lower layer. Or more accurate, they address the way individuals work with this lower layer. If you imagine now my organizational chart was Scrum encompassing say there's a big circle around management and individual contributors and then another overlapping circle that encompasses individual contributors and the technologies they use. That's how I see Scrum and DevOps is working together.

With that in mind, let's talk a little bit about what DevOps is and maybe where it came from. I don't know how many organizations did it this poorly, but I've certainly worked in and I imagine many of you have worked in organizations where you can at least see parts of this pattern emerging. Where the developer's goal is to release software as quickly as possible to meet customer needs. Whatever software you're building it's not useful until customers can use it. As a developer, the goal is to make that software do things customers need and to make those changes as quickly and as responsibly as possible.

Then you also have this idea that the operations team, the people who run the servers and the network, their goal is to make sure that the network and the system keeps working and works reliably and that it doesn't crash or if it does that, it can be restored quickly. That it operates reliably, that it operates quickly. This all sounds great. You want peoples to make software that customers can use quickly. You also want people to make sure the system operates smoothly.

The problem is that when you frame it like this, you're actually putting developers and operations at the opposite end of a tug of war. They're aiming for opposite goals this way. Your developers, if you boil that whole concept down, they are striving for rapid change, and operations, if you boil their whole business purpose down, is to prevent change. Stability and change are opposite things. You have your developers and your Ops people essentially fighting with each other. That doesn't mean that they literally break out into fights or yelling matches, but their goals are in opposition to each other.

The idea of DevOps was born out of the realization that this is probably wrong. That having our developers and our operations people at odds with each other is probably not the best approach toward a holistic and functioning organization. I like to simplify the idea that DevOps just means cooperation. Now, there are many flavors to that. If you just think of that as a shortcut, that DevOps is a cooperation between development and operations, it can cut through a lot of nonsense out there in the literature, especially advertising literature, from companies trying to sell their DevOps products. Just remember that DevOps means cooperation. It can cut a lot of things out in terms of confusion.

There are different models or different ways to look at DevOps. One of the most popular ones is the stages of DevOps. You may have seen this in the past. If you go to Google image search, you can find the infinity loop that is the stages of DevOps. It's two circles conjoined with arrows pointing so that the flow always goes in one direction. The stages of DevOps are plan, create, verify, package, release, configure, and monitor.

Now, I'm going to go through each of those in a little bit more detail just to give you a clear idea of what each step or each stage is about. Keep in mind, though, that these stages are not necessarily distinct. That is to say that some stages blur or merge into others, some stages may happen simultaneously, depending on the team or the technology you're using. The point here is not the chronological progression of these stages, but rather to think of each stage and make sure that it's happening in your team.

The first one's planning. That's hopefully fairly self-explanatory. This is also the stage that has the most overlap of Scrum. Scrum has your sprint planning, backlog grooming. These are all parts that can happen or influence the planning stage of DevOps. Once you have a plan in place, then you can create and that's the part where developers do most of their work writing code or you can have even your copywriters or your graphical designers doing work there. Verify that could be going to a QA team for verification and sign-off. It could also include and often includes both a QA team and some sort of local unit tests, maybe you're doing TDD or developers writing their own tests in some other way.

Another is packaging. Packaging is, of course, when you compile everything, maybe do some minification or compression, build a package that can be distributed. Then you release that eventually. That can take many forms. It could be, in some cases, sending physical media such as a USB stick, or CCD or DVD to a customer. More common these days, it means pushing to a server somewhere so that you can go live.

Then configuration. Depending on the type of software you're doing, configuration may happen even during or before packaging, or it might be something that the end-user configures or a combination of both. Of course, for applications like say a mobile app, the configuration typically happens by the end-user after it's been installed. For more web-based, maybe a SaaS type thing. There's probably some configuration that's handled by your operations team and maybe other parts of configuration handled by the end-user. Configuration can happen in various stages along here.

Finally, monitor or monitoring. This, of course, is when we make sure that the system is working as expected. That can be a simple thing, such as going to the website to make sure the website's up. More often it's going to involve some sort of automated tooling to maybe check the website every five minutes to make sure that requests aren't taking too long. It could involve all of your log output. There's a whole lot that can go into the whole monitoring of application. Then that feeds back into the next iteration of a plan. That is a very brief overview of the stages of DevOps.

There are, however, a lot of misconceptions and myths about DevOps, and I want to address some of the more serious ones here today. One common misconception about DevOps is that it is a set of tools. If you use a CI/CD pipeline, or you do some application monitoring, then you're doing DevOps. That's just not true. Of course, people who do DevOps will be using tools to accomplish DevOps.

The same way you can't say that, "I bought a camera, therefore, I'm a photographer," or, "I have a paintbrush, therefore, I'm an artist," you can't say that just because you have certain tools that you're using that you're doing DevOps. DevOps is more like a mindset or an application of tools just the same way that art is more of a mindset or the application of tools.

Another misconception about DevOps I hear a lot is that if we use DevOps, then we ought to be generalists. That is to say that I don't want to do DevOps, because I don't want to have to learn back end. Or I don't want to do DevOps because I don't want to have to learn Kubernetes. Or if we're doing DevOps and everybody's writing code, and everybody is deploying code or whatever, that's just not true. There's absolutely nothing about DevOps that says, "You can't have specialists."

Remember, DevOps is about cooperation. There's no rule in cooperation that says, "If you cooperate, everybody has to have the same skills." That's just ridiculous. It's ridiculous with DevOps, too. It's absolutely okay, and it's usually best to have dedicated developers and dedicated operations people, even if you're doing DevOps. DevOps simply means that Dev and Ops work together in a cooperative manner, not in silos, not in isolation, where their goals are in conflict with each other.

Another misconception I hear sometimes is the idea that, "Because we're doing DevOps, we don't have any rules." Or, "I'm a developer, but we're doing DevOps, so I should have root access on all the production servers." That's silly. DevOps doesn't mean a free for all, it doesn't mean that all developers have complete operational access to everything. In fact, it often and probably usually means there are more rules and more strict controls.

As an example, one of the rules I follow when I'm working on a team with DevOps is that no human from Dev or Ops or anywhere else should ever log in to a production server. We do this for the safety it provides. This forces us to go through some sort of automated process to update configuration or software on a production server. DevOps often means more rules, but think of those rules as guardrails to provide safety, to allow us to move more quickly.

Lastly, DevOps is not a job title. I know that a lot of recruiters on LinkedIn would disagree with me but DevOps engineer is just an oxymoron. If you have an engineer dedicated to DevOps, they're literally not doing DevOps anymore. DevOps is about not being in silos. If you have a DevOps team of DevOps engineers, you're putting them right back into that silo they were supposed to be coming out of. DevOps is a mindset, it's not a role.

The same way that Agile-- I don't think anybody would ever say we're hiring an Agile engineer, except to use agile as an adjective. They wouldn't say your title is Agile engineer, you're the one who's doing agile on our team, while the rest of us go back and do waterfall or something like that. That's just utterly ridiculous, yet, it's exactly what so many people try to do with this DevOps engineer title.

I challenge you to go to LinkedIn, do a search for DevOps engineer, and look at the job requirements. I'll bet you that the vast majority of them have these sorts of requirements. They're going to be doing software packaging, software release management, configuration management and monitoring. These are the exact same things that fall under operations title. There's no software development in there, there's no verification. There's very little planning, or at least not planning of the software features, maybe of the infrastructure. Every DevOps engineer is actually an infrastructure engineer that I've seen.

Get rid of the title, stop calling things DevOps engineers, call them an operations engineer, if you want, or an infrastructure engineer, or a site reliability engineer is a good title too although it has slightly specific connotations. One other thing, of course, doing DevOps alone doesn't make you agile. You could, in principle, do DevOps on a waterfall team. You could do all of those stages in sequential order with big design upfront and standard testing stages, and so on so forth. It will probably help your waterfall project be more effective, but it won't make you agile.

Now, having said that, I believe that to be truly agile, you need to be doing DevOps. Maybe you don't call it DevOps, but you're going to be doing the same things. If you want to have an agile release cycle, so that you're delivering software quickly, DevOps is what gives you that capability. I mentioned the seven stages of DevOps. There's even a higher-level abstraction we can look at, which we call the three ways of DevOps. These are, if you will, almost the manifesto for DevOps. It's not nearly as formalized as the Agile Manifesto.

If you look at these three ways, and these three ways are talking about a lot in the book, The Phoenix Project, if you haven't read that it's a good fictionalized narrative novel about DevOps, or the more recent Unicorn Project which is about the same concept but more from a software development standpoint.

Well, let's talk about those three ways. The first way is that work always flows in one direction. This is built into the infinity sign that I described earlier. In that in the infinity sign, the arrows all point in one direction and then they loop back to the beginning. What this means in a software project, typically, is that you should, of course, never pass known defects down the stream. If you're writing software, if you're writing a new feature, and you know it's not ready or it doesn't work right, you should not pass that downstream. You need to fix or remove potentially those bugs. Maybe that means removing a feature until it's ready before you continue down the process.

Then you also minimize or eliminate upstream flow. This goes back to the whole tug of war I was talking about at the beginning between developers and operations who maybe fighting with each other. We want to eliminate that. When you have a developer say, "I want to release this software package," and the operations guy looks at it and says, "Oh, but there's a bug," or, "It doesn't look stable enough." Know that that's that back and forth that we're trying to avoid here.

Work should flow in one direction only, and it should not go upstream. Then once we have that in place, then we should seek to increase flow. Once we have all of our work going in one direction, we don't have our work ever going back to the direction, then the next optimization is to get more work going downstream.

The second way is to create, shorten, and amplify feedback. I just said that work should only flow in one direction. However, information should go the other direction. When I pass completed work down the stream, say from the create process to the verify process, or from verify to package, that appropriate, but each of those stages should have the ability to report back upstream about what's going on. The simplest example of that probably is monitoring. If only your operations people have access to say logs and monitoring metrics, that is not sending this information upstream.

As a developer, I have written a new code feature or tried to fix a bug and I deploy that to production, I should be able to immediately see those logs and whatever monitoring is available so that information can be communicated upstream to me, so I can see how my feature or my bug is working in production and potentially either sign it off as completed or provide another tweak if I need to do some more work on it.

Then once that information is able to feed back, we want to work on shortening those feedback loops. What that means is that, at every reasonable opportunity, we want to give information upstream about what's going on. Let me offer a simple example. As a developer, I'm working on some code and maybe I'm using TDD or some local tests of some sort, those can provide a very quick short feedback, but on a limited scale, obviously, right?

My unit tests may validate that my function does what I think it does in isolation, but that's not complete, that's not the same as going through a QA stage. Then I pass it along to verify, maybe a QA engineer does the manual tasks. Maybe it goes through a larger test suite with some end-to-end test or integration test. As soon as it discovers there's a problem, it should feed that information back to me.

Once it's past that stage, it should go into the packaging and release and deployment stages, and then it gets deployed to production and then we should have some monitoring in place that alerts me ideally as quickly as possible if there's a problem. Maybe I made it through all the test stages, but still some defects got through that prevents the server from behaving correctly. I want to get that information back in an alert or some alert or notification as quickly as possible. Then want to amplify these feedback loops. That just basically means make important things loud. If the server goes down, we should know about that quickly. If it's maybe less important, maybe things are just a bit slower than they used to be, maybe that doesn't need to be quite so loud.

Finally, the third way of DevOps is to experiment, to learn, and to achieve mastery. This should not be new to anybody doing Scrum, because Scrum is in many ways about applying the scientific method and experimentation to software delivery. Turn everything into an experiment. Always be looking for new experimentative ways to improve your process. Of course, improving your software is great too, but as far as DevOps is concerned, it's more about improving the process of software delivery.

Learn from your success and failure. When you have failures and they do and will happen, make sure you're learning from them, do some sort of post-mortem or retrospective and learn from these failure failures so that you can improve in the future. Achieve mastery through practice. It goes back to the old agile saying, "If it hurts, do it more." Same applies to DevOps. That the more you do it, the more you release, the more you set up monitoring, the better you'll get at it.

We're coming to the end here. How can you get started with DevOps? The first thing to recognize is that that change of any sort is difficult. There are no easy answers. You're not going to just flip a switch and be doing DevOps. Also, DevOps just like Agile and Scrum is really a big topic. There's always more to learn more. We're always learning more. Every year we have more data and more new ideas on ways to do DevOps and to do software delivery better. This isn't something you can just do and be done with it. It's a lifelong, career long journey. That doesn't mean you need to become an expert before you can start benefiting from it, but it does mean there's always something to learn. Even so, it's important to take that first step. You can't start learning from it until you start.

One of the easiest ways in my view to start with DevOps is to start with continuous deployment. In a nutshell, continuous deployment is simply the idea that changes are immediately deployed on merge. Think of it this way, you hit the merge button using GitHub or GitLab or something like that. As soon as your developer hits the merge button, it kicks off some scripts in the background that build and deploy your software in front of customers, real live actual customers, with no further human interaction.

Ideally this takes 5 to 10 minutes. Sometimes it can take longer. It might even take a few hours depending on how complex your software is but as quickly as possible most projects I've worked on. It's been 20 minutes or less to do this. This can sound scary and often this feels like it should be an end goal rather than a starting point, but I want to make the case that this is a good starting point.

Why? Well, perhaps the most important reason is that you get a shorter development cycle. If your developers can push the merge button, another code is in front of real customers within say 20 minutes, they don't have to wait for feedback from the customers. You don't have to wait until the next three-month release cycle is over or something like that. It feels risky but it actually reduces risk. When you release each feature individually, the aggregate risk is reduced.

If you have 10 developers working on code over the last sprint and release them all together, and something goes wrong, then you have to figure out which of those 10 changes caused the problem. If you release each one individually, it's much easier to pinpoint, "Oh, this problem started at 3:00 PM on Tuesday, that means that code that Bob merged at 3:00 PM on Tuesday is probably the cause of the fault." It's much easier to debug and isolate the problem that way.

It also adds a sense of responsibility. When a developer knows-- and I say this from personal experience as a developer myself. When I know that I am about to push something to my customers, rather than to a review queue that might get released in a week or two, I have a stronger sense of responsibility. I feel a slightly bigger, in a good way, a bigger weight on my shoulders and I maybe think a little bit harder about, "Have I tested this thoroughly? Am I certain this is correct?" before I hit that merge button.

How can you do this if you don't have all of the fancy automation that DevOps people often talk about in place? Well, this is what I call lean continuous deployment or lean CD. It's the idea that we can do continuous deployment without any automation in place, specifically without automated tests in place, which is usually what people are talking about. It's actually technically easier than you think. I have talked about this concept even recently with some very seasoned DevOps people and they look at me like I'm crazy, but trust me, it's not as difficult or as crazy as you think. Bear with me here.

If you can get lean CD in place, it serves as a starting point for continuous improvement. It's not the end goal. Fear is always the biggest obstacle I hear when talking about this. The basic idea with lean CD is, you keep doing all of the manual steps you're doing now. You have manual QA, presumably, maybe you have a product owner sign off before the change goes live. You keep those things in place. You just do them before you hit the merge button.

Now, this might seem a little bit overwhelming. It's not that complicated. I have a free email boot-camp that will walk you through the steps. It's a 10-day boot-camp. Of course, actually doing the steps probably will take longer than that unless you have a very simple project, but go to leancdbootcamp.com and you can sign up for that free boot-camp and I would love to hear any questions you have. Just reply to any of those emails. It will come directly to me and I'd love to hear back from you.

Thank you for listening to my presentation about how Scrum is not enough and how DevOps is essential for Agile success. If you have any questions about the material today or any other DevOps related topics, I'd love to hear from you. Send your questions to jonathan@jhall.io. If you're brave enough and you want to send a video or an audio recording, that will go to the top of the list, but plain text email is fine too and I'll try to answer these questions in an upcoming episode. Until next time.

[music]

This episode is Copyright 2021 by Jonathan Hall. All rights reserved. Find me online at jhall.io. Theme music is performed by Riley Day.

[00:30:53] [END OF AUDIO]