Resources:
Book: Working Effectively with Legacy Code by Michael Feathers:
Martin Fowler's blog
Legacy Code Rocks
MenderCon

Today's Guest:
Peter Morlion
Personal tech blog
Technical debt blog
Twitter: @petermorlion
LinkedIn: petermorlion

Transcript

Jonathan Hall: Ladies and gentlemen, The Tiny DevOps Guy.

[music]

Hello, and welcome to episode number four of Tiny DevOps. I'm your host, Jonathan Hall. Today I have with me Peter Morlion who loves fixing and improving legacy code so the next software release can be less stressful. Peter, welcome.

Peter Morlion: Hi, thanks.

Jonathan: Can you go ahead and tell us a little bit more or elaborate on what I just said? What's your fascination with legacy code? What do you do professionally and your history?

Peter: Sure. Actually, if we go way back, I'm not the kind of kid that grew up programming. Actually, I'm fairly late to the game. I did like computer games but when it was time to choose go study, I chose political sciences. Only later, I did computer science in night school and there, I discovered I like programming. Then when I was looking for a job, I just took the first opportunity that I could as a programmer. It was working on an existing application and so there was a lot of refactoring, there was a lot of writing new code and new features but there was also refactoring.

There, I started noticing that it's very pleasing to after like a year or maybe two years, you start really knowing the application, the ins, and outs, the old quirks. There, I started noticing I don't really mind what they call brownfield developments or the legacy code, refactoring legacy code. Along the way, more and more, I got new jobs and went from place to place and there was always some refactoring going on more than there was new code.

While many of my colleagues hated legacy code and just liked starting stuff from scratch, I just noticed, "I like this stuff, so maybe I should just try and make a career out of it or at least focus on that." More or less coincidentally, I just found out that was fun, so that's how I got into liking legacy code.

Jonathan: Excellent. That's a nice story. I agree. I think most people try to run away from legacy code. I also am drawn to it, I feel like it's always a fun challenge to try to make the code better without changing the functionality. It's an interesting constraint to work on. Today we're going to talk a little bit about technical debt but I would like for you if you don't mind to define that term. You talk about legacy code but what is technical debt to you?

Peter: Great question. Everyone or at least all developers seem to know what legacy code is but asking a definition makes it a lot harder. It's like, "Oh, it's spaghetti code or that doesn't work." Digging into this, there is the definition by Michael Feathers who says it's code without tests but I believe that we should see that more in light of when it was written. It's from his book, I forget the name of the book.

Jonathan: I think it's Working Effectively with Legacy Code.

Peter: Yes, that's the one. I think it was written in a time where unit tests and automated tests weren't as normal as they are now even though I've read quite some codebases where it still seems something new. I tend to disagree with that. I don't know, maybe Michael Feathers himself disagrees with that now because I have encountered code with a good test suite with quite some coverage but where we still feel this is legacy code, has a lot of technical debt, it's a mess, it has to be improved.

A good one I heard recently is legacy code is valuable code you're afraid to change and that covers the emotional part with legacy code. We have some code and it's valuable, either it's making money, or it's just useful. We still need it even if it just doesn't create income, but we're afraid to change it. Each time when we need to change it, we're afraid we're going to break things or we just don't like it because it's difficult to work in. That's a good one.

Another very good one is by Andrea Goulet from Corgibytes and Legacy Code Rocks. She says legacy code is code without sufficient communication artifacts to explain its intent. The first one is more about the emotional aspect, we're afraid to change it. The definition by Andrea is more a technical definition because it explains, it might have tests but it might be lacking documentation which still makes it hard to work with, or it might have thousands of pages of documentation but it's outdated.

The fact that she talks about communication artifacts, that can be anything. That can be documentation but it can also be code commons, it can be unit tests. I think it's a great definition because that's usually what we feel calling code and we don't know what is this code supposed to do, what is it doing? We have to start digging and digging and reading and making a mental model. Those are two great definitions to me.

Jonathan: That's good. What are the risks of technical debt? We have some code that we're afraid to change, how does that affect us especially from a business standpoint? If maybe I run a business and I don't rely on the software, what are the risks involved?

Peter: Yes, that's the core of it because even if we are afraid to change it, that has no business impact but I believe legacy code can have a real business impact in many ways. The obvious one is each time we change something, we create bugs that we didn't know we were creating. We fixed something here and then created a bug somewhere else, which can have an impact on the user base of the company.

There's probably some way of quantifying that. It can mean money lost. There's also time lost. Each time a team or a developer needs to add a feature or change a certain feature. If it's difficult code to work with or they're afraid to work with it, they're going to work slower than if it had been good clean code, easy to read, and easy to maintain.

Time lost means money lost as well because it's an opportunity cost. Developers could have been working on the next feature but instead, they're working days and days just add a button to something that seems fairly simple. There's that. There's also less obvious things like security. You might be stuck on an older platform or on an older framework or library which has security issues now and might even be impossible to operate because of other dependencies. You can get stuck in an old version and you can have all kinds of implications.

Another thing is it's hard to get to know the application, so there's slower onboarding. If a new member joins the team, it can make a difference between getting up and running in two weeks or only being productive after six months. The final thing that is often forgotten because we tend to look at problems in a technical way but there's also again the emotional aspect. Too much legacy code and technical debt leads to burnout in the worst case but just can decrease the morale of the team and worst case, people just come and leave after a few months.

I have seen this in action where there's a constant change in the team because people don't last longer than a year, which is bad for business again because you have to onboard the new developers each time and people have to get to know the application notification. They'll more easily introduce new bugs by accident because they assume certain things or they don't assume other things.

Jonathan: Yes, that's good. You've brought up some ideas, some concepts that I hadn't actually considered. The emotional aspect especially of technical debt is something I've-- If someone had asked me, I probably could have said that but it wasn't top of mind for sure and so that's really important.

Is it ever appropriate to have technical debt, to make a choice that we're either not going to refactor this thing, we'll leave this legacy code as it is or we're going to maybe even introduce legacy code? I don't know if you would intentionally introduce legacy code as you've defined it. We often choose as a business or even an individual to take on financial debt to buy a house, a car. Is it appropriate to make that choice as you define legacy code and technical debt or should we always avoid it?

Peter: Yes, I think there can be situations but you have to do it deliberately. There's this categorization, there's intentional debt and unintentional debt. Things just happen. Even if you're putting all this energy and making good documentation and tests and clean code and well-factored and all, inevitably, dependency, a security issue maybe you didn't know about it until a year or two later, or things that were clear to the team at a certain moment might not be later and so they didn't think to document something.

Or new patterns emerge like the Singleton pattern when I got started as a developer was one of the design patterns you had to know. Fast forward a few years, and it's considered a bad practice except for maybe some edge cases. You might end up with code at 4 or 5 or 10 years later, even though at the time, we felt good with the code, our feelings have changed now and we don't think it's readable or maintainable or good code. That's unintentional code and that'll happen inevitably.

Sometimes it is a good idea to create intentional technical debt because there's a deadline in the project so things have to go fast, or maybe you haven't found a better way so you're writing code and it works and it's under tests. You've tried refactoring it but then it doesn't work anymore, and you're sort of stuck. Then it's sometimes it's a good idea to choose to leave it as it is and revisit it later. The problem there is that often revisiting later never happens.

In the end, we're still working for a business or maybe we're running a business so there's more than just the pressure to write good code. There's also pressure to get something out, there's a pressure not to waste too much money on things. There's all these different factors. Sometimes it can be a good idea to take the quick route and create some intentional technical debt.

Jonathan: Do you have any examples of a specific time when technical debt was a serious problem?

Peter: I have a great example, it wasn't great to work with. There's this company which I won't name. They have a big application for making reservations, for booking trips. It's in the tourist sector. It's a gigantic application. I believe it was a monolith but it existed from seven or eight pieces. It had about eight databases. It had, I guess, between five and eight teams as well, all working on that.

Management acknowledged, "Okay, it is a mess, we're stuck, and we need to fix it." They got in some contractors, some freelancers to create a refactoring team which I was part of. Basically, it didn't work out because you can't have one team saying, "We'll do the cleanup," and the others just chugging along. The issue there was again, each time business or the business team would ask a feature, the relevant team, one of those five teams would get to work and it would take a long time.

It would usually be riddled with bugs. This wasn't a fault of the team, there were some capable people there, seniors and juniors, but you have to imagine there was a lot of business logic in SQL scripts. They had stored procedures referencing other stored procedures, referencing again other stored procedures. Many of these stored procedures were thousands of lines long, so to find your way into that even for a small change is just frightening at best.

When you make a change, you don't really know, especially as newcomers because we, the refactoring team were new. You don't really know what you're doing because you might be editing it in the wrong place, or you might be doing it in the right place but there might be two or three other places where you need to make the change. This had been going on for years probably, and a lot of developers just left, especially the better developers who knew, who saw this and saw there wasn't going to be any real change.

They just left for something better. What you're left with is people either choosing for the cushion job, or you get stuck with developers who don't have enough skills to really change the project or turn it around which is the dead sea effect. I think after six months, just about everyone in the refactoring team just left. [chuckles]

Jonathan: Technical debt can be scary. How do we know if technical debt is affecting us? Maybe you've just joined a new team especially and you don't know the details of the company and the codebase yet, how can you tell? What are the signs to look for?

Peter: Of course, there are tools that can measure this. I know in the .NET space, there's NDepend, which actually just gives a score of technical debt, and even can calculate more or less how many hours or days of work you have ahead of yourself to fix it all. There are also other metrics like code coverage culture, and that there's all kinds of things that you can measure to see how bad is it in my application.

There's also just talking with the team, again, that emotional aspect, seeing if people like working on the code or if they hate it. Maybe count how many books there are after each release or you don't even have to count it if there's a lot, or people feel it's a lot. Maybe there are only five after every release but there are five big ones each time, and people feel this is way too much.

Just the feeling of we're facing an issue here that needs to be fixed, that could be a way of measuring that. I have this metric of myself which I call the disaster release ratio. You count how many disasters or bugs you have after every release and divide that by the amount of releases you have, and you want to get a smaller number as possible. If you done five releases but you had 25 bugs over that period, the ratio is five.

If you have one bug and you released it five times, it's 1/5, so it's a lower number. Just getting a feel of that is a way of measuring.

Jonathan: Let's say that we've identified that our team is affected by legacy code, what can they do about it? How do we start tackling this problem?

Peter: Writing tests is to me, the number one technique to tackle your legacy code. At least it's a starting point. When we encounter legacy code and technical debt, we probably want to change the code and improve it. Before we do that, I believe we need a way of knowing that when we make changes, we aren't breaking stuff because often in legacy code, things are constantly breaking and that's one thing we want to stop. Adding automated tests is just about the starting point for me.

Sometimes to add automated tests, you might need some little refactoring or tweaking before you can even add the tests, but I'd keep that at a minimum which is why sometimes I wouldn't-- There's a big debate, the test pyramid where at the bottom of the pyramid, there's unit tests, so you need a lot of unit tests because that's the base. The more components you start testing together, the less of those tests you need. Usually, you need a lot of unit tests, a little less integration tests, and even less end-to-end tests.

With the legacy code, that's often or sometimes, at least, that's not as easy because adding unit tests requires a certain design maybe so you can inject dependencies or mock certain stuff. That's not always as easy. I've had projects where I've done it more or less the other way around where I started writing a lot of end-to-end tests even though you get then the, I think it was called the testing ice cream cone or something where it's the inverted pyramid but you make do with what you have.

Having a lot of automated tests, regardless of the type just gives you a safety net to start refactoring. I actually have a success story there. At another client, we had a similar-- it wasn't as bad but we had a similar issue where this application, each time things changed and were released, something would break. The first thing we did, we started writing a lot of postman tests.

Basically, we wrote as much as we could for that, and then when we felt we had enough things covered, then we started changing the code and we could run our postman collection and see, "Oh, I've broken something, let me debug that or revisit that." It really helped. That is the starting point, writing automated tests so you can change code, run the entire test suite maybe in CI build and verify that you've reduced the risk of breaking anything.

Another technique, which I haven't used yet, but I find a very interesting idea is the wall of technical debt, which is an idea by Mathias Verraes. His idea is when you encounter something that you feel this is an issue, this is a problem, you write it down on a post-it, and you put it on the wall. Everyone does the same. If you encounter something that's already on the wall, you can add a dot notation, just add a dot and dots will be added on the post-its as you go along. That starts making visible which pieces you need to tackle first. You can add estimations on how much work it would take to improve this code.

Then it even makes it visible to business people which pieces, or which features, or which pieces of the code are a problem and how long would it take? Then you can start evaluating, this is low-hanging fruit. It won't take a long, but it seems we're very frustrated by it. That's a good idea then to start picking which pieces you should be working in first and probably also, which pieces should we be writing tests for first and then start refactoring, get agreement on the team on how you want to refactor.

Maybe variables need to be renamed in a certain way. Maybe there's coding styles, maybe there's architectural styles, maybe you want to introduce the repository pattern. These are things you need to agree on in the team. Once you have that, you can start working towards that.

Jonathan: What can we do if maybe management hasn't bought into this idea? I think to some extent, practically, everybody's in that situation. Maybe even management agrees, "We had technical debt," but they never want to give it the full attention that developers want. What could we do? If we are feeling the pressure of technical debt in a way that our project manager or upper management is not, what can we do?

Peter: The wall of technical debt, which I mentioned is a good idea to make it at least visible. You could even, I think Mathias Verraes mentioned that. If each dot on the post-it represents half an hour, you might even make it quantifiable where the business or the manager can see, "Okay, the team estimates this takes, for example, five hours to refactor, and they've spent 50 hours on it. We're just losing time, maybe just refactor and get it over with."

That could make it visible and maybe help you convince them that it's in their interest also to refactor because just complaining about clean code and technical stuff often doesn't resonate with business people. I think what you need is find a common ground to explain what the issue is and why we need to refactor. There's the idea of technical depth explaining, "We're taking on debt like financial debt, and we need to repay or things will just get worse in the long run."

Although, there seems to have been some other ways of looking at that recently because business people are usually used to taking on debt, and it's something they can plan for, and it's fairly normal. There's other metaphors also in the financial space where-- now I'm not a financial expert, but someone compared it to an investment that isn't hedged. If I understand correctly, you can invest in say stock or shares, and thinking things will increase, but then you never know what the market is going to do.

The shares you bought might decrease in value and then there's some way of insuring yourself against that, which they call hedging your investment. Of course, you pay for that as well. If the shares increase in value, you'll win money, but you'll win less, or you'll have less profits than if you wouldn't have hedged the investment because you have to pay for hedging that investment. If they lose in value, you won't have lost as much. It's insurance and you pay for insurance as well. You could compare technical debt to that as well.

That resonates, at least that's the theory of the author of that article. That's resonates better with business people because now we're talking about risk, no longer about debt, which is something they can plan for. They can manage which is foreseeable. We pay this amount every month. While risk is something that people don't like, they want predictability and a debt is something predictable, but a risk isn't. That might resonate better. In general, I think-- I need to think more about this, but I think we need to find ways of communicating on a common ground.

Maybe even if you're talking with some manager who loves doing marathons, you could compare technical debt with starting fast at the start of the marathon and running very fast. Then, of course, you still need to do 30, 40 kilometers. You'll have a hard time, and you might even have to give up while if you just find a pace and you can continue that pace for 42 kilometers, then that's probably a better idea. We need to get out of our technical comfort zone and find connection with business people so that they understand why we're complaining about clean code.

If all else fails, I would sometimes recommend just refactor anyway. Take ownership of the code as a developer, or as a team. Businesses asking you to implement a feature. They don't, and they shouldn't really care how you implement it. If I need some plumbing done in my house, I don't care how the plumber does it. It just needs to be fixed. If there's a leakage, just fix it and do it according to the rules of the game, or what's the best practices of plumbing.

I think business can expect the same from us, that we implement a feature according to the best practices of our trade, of our profession. If they're pressuring us for delivering by the end of the week, we should be prepared of pushing back, or just when you find time in between, refactor anyway.

Maybe I shouldn't say this, but I have done this where a technical lead told me not to spend time writing unit tests, and I just did it anyway. That's just the way I feel comfortable writing code.

Sometimes we should just take ownership for the code and say, "We're going to do it right, and good. We're going to refactor this piece." You don't need to ask the business, "We need four weeks to refactor this." You need to do continuous refactoring, they call it. you need to do it continuously bit by bit, baby steps. You don't need to ask and it's unrealistic to ask business for like a week or four weeks to, "We won't be delivering any business value, we'll just be refactoring stuff," which in the end does create business value because it makes your work easier and more efficient. It's unrealistic to ask it from them.

Jonathan: Good advice, good advice. Where can listeners go for more information if they're struggling with technical debt, and they want to learn more about what we talked about?

Peter: There's the Working Effectively with Legacy Code by Michael Feathers, which give some great also technical techniques like extract classes and interfaces, and very, very specific ways of improving your code. A lot of the blog posts by Martin Fowler also cover topics on improving and changing code. If you like working in legacy code, then I can recommend Legacy Code Rocks or the Legacy Code Rocks Community. There it's Slack, they have a podcast, they have online meetings. You can find them at legacycode.rocks.

I believe they also organize MenderCon, which is like a online at this moment conference on legacy code. They have this idea there are makers and there are menders in programming. There's a place for both. There are programmers who like making stuff from scratch, and there are others who like taking something that has been built and improving on that. Even if it's in a bad state, get it in a good state again. Those are the menders. It's actually Andrea Goulet, which I mentioned in the definition who is one of the founders of Legacy Code Rocks.

Jonathan: Finally, how can people follow up with you if they're interested?

Peter: You can just google Peter Morlion because it's a fairly uncommon name and you'll probably find me. I'm on Twitter at @petermolrlion. I also have two blogs, which might seem strange, but I've petermorlion.com where I just write about anything, technical, usually, but anything I want. I have redstar.be, which is my company, where I specifically blog about legacy code and technical debt.

Jonathan: Very good. Thank you very much, Peter, for coming on today. It's been an interesting and educational conversation. I hope the listeners have gotten something out of this. Thanks so much.

Peter: Thanks for having me.

[background music]

Jonathan: This episode is copyright 2021 by Jonathan Hall. All rights reserved. Find me online at jhall.io. Theme music is performed by Railey Day.

[00:31:04] [END OF AUDIO]