What is the best/ethical way to estimate code clean-up?

Question

I have been working on a product line since my senior year of college for my current employer. Now several years out of college I am in a position where I am giving estimates for change requests on this same produce line. Please note that our software lead still approves the estimates, but is hands off during the process.

As my skill set continues to grow, I continue to see things which I can improve upon. The issue is how to get these changes funded. My current process is to clean up / refactor code if it touches an area that has an active change request. However the dirty sections of code that aren't part of an active CR are neglected and bug me more and more as the product line matures.

To help clarify, I do not believe that the customer should pay for a full rewrite of the software, unless it is in fact cheaper to do so to meet new requirements. I do prefer the incremental changes to clean up bits of code.

When submitting quotes / estimates, what is the best way to ethically try and recapture the costs of code clean up?  

To elaborate further, what kind of padding is typical for general maintenance? Should it be based on lines of code, project complexity, or some other metric?

Answer 1

The business problem of technical debt

The problem I see here is more of a business problem with technical debt rather than a technical one. I'll answer your question with another pertinent one.

• Why should a customer pay you to rewrite code which will do the same job as it did before, whilst they gain no direct feature or financial benefit?

Difficulties of acknowledging technical debt to customers

Trying to get a customer to acknowledge technical debt is difficult at best and more often than not ends up becoming a case of "why did you write it that way to begin with if it's going to cost me more now?".

Being aware of technical debt and fixing the technical debt as you touch parts of the system incrementally as you have been doing is pretty much the way I would recommend doing any large scale debt reduction.

For any new work I also recommend padding estimates taking into consideration for incrementally fixing your build. It's natural to assume that the more complex the software becomes the higher cost of maintenance is involved. Refactoring and reducing technical debt is an active part of maintenance that should be paid as soon as possible, however break up the maintenance costs into smaller more palatable pieces for your customer.

What Robert "Uncle Bob" Martin says about Ethics and Software Craftsmanship

I think Robert C Martin gives this same recommendation about tackling this very topic in his talk in Part IV. Sorry it's a long watch but well worth it.

• http://www.infoq.com/presentations/craftmanship-ethics

Edit

Metrics for calculating technical debt is pretty much the same as for estimating time to perform paid work. The trick is identifying the pieces that give the most pain, and chip away at them gradually. Always refactoring so that they always work, however slowing becoming nicer to work with and easier to maintain.

A real world example

I have a backlog of technical debt at the office of a piece of software which has a DAL which is about 80% NHibernate2 but the legacy portions uses Stored Procedures and manual proxys with domain logic written into the SP... (nightmare).

However for every maintenance project we may do say 10 maintenance tickets per 2 week sprint, 1 or 2 of those tickets will be a technical debt item. And the customer is happy with this as these are maintenance items and we get them to sign off on them as part of the maintenance sprints.

So effectively break each technical debt item up into a ticket which could be worked on by 1 developer for a 2 week sprint. This will give you a metric as to how much will it cost the customer, and also give you an indication of velocity as you go through a couple of sprints you will see progress.

Answer 2

The goal of refactoring is to make code cleaner, to decrease the cost of maintenance and extensions in the long run. Since it is very difficult (although not always impossible) to get management or clients understand the concept of technical debt, it is better, and is also more practical, to do it in small increments, preferably in association with some necessary code change. So whenever you need to touch a piece of code, you also make it somewhat cleaner, in proportion with the size of the concrete task. I.e. if you need to change a single line of code to fix a bug, fully refactoring the containing 100-line method is probably out of proportion. But e.g. renaming some local variables to make their purpose more obvious, or extracting a couple of smaller methods to make the parent method easier to understand is fine. Thus, out of a, let's say, 3-hours effort to locate the root cause, write unit test(s), fix the bug and run all regression tests, half an hour spent on refactoring is fine, two hours is not :-)

OTOH if there is code which - however ugly or old it looks - hasn't been touched for a long time, works well and doesn't hinder the maintenance of the rest of the program either, it is probably better to just leave it as it is, however nice it would feel to ourselves to have it as shiny clean and perfect as the rest of the code (I know exactly how you feel, been working on legacy projects most of my career :-). But as doing so would not have any effect on maintenance costs (since there is no feature request related to it), bug rates (since there are no known bugs), it would provide no benefits for the client.

All in all, in real world projects, our refactoring resources are limited (often severely), so it is best to focus them on areas where we (and our clients) get the best return out of investing our effort and time. This, in turn, is also the best way to prove the value of refactoring towards clients / management, which may, in the long run, help you get more time for it...

Answer 3

I see it as a necessary part of maintaining the application. For an ongoing project I don't think it's unacceptable to factor into enhancement quotations an amount of time for re-factoring. Not loads and loads, but as the size of the project grows, maintainability grows. You can always justify it as well: you are in fact saving the client money in the long term as you are avoiding (or at least prolonging) the need for a re-write.

If you go to the garage to have a part changed on your car, and the mechanic notices an issue with something small (a broken nut or what-not) he will fix it. He won't charge for it, it's a part of his professionalism. However, his initial quote to you will have a bit of padding to insure himself against finding these things, against coming up with problems etc. It's why quotes for older cars are more than for newer cars: people rarely refactor their cars to keep them going longer until they break.