How can I avoid always feeling like if I completely rebuilt my program from scratch I'd do it much better?

Question

I have learned a significant amount of coding, however, it's always been in a scientific environment (not computer science), completely self-taught without anyone to guide me in the right direction. Thus, my coding journey has been ... messy. I've noticed now that whenever I build some type of program, by the end, I realize how I could have done it far more elegantly, far more efficiently, and in a way that is far more flexible and easy to manage going forward. In some circumstances, I've actually gone back and rebuilt things from the ground up, but usually this is not practically feasible. While most of my programs so far have been relatively small, it seems quite unwieldy to completely rewrite large programs every time you create something.

I'm just wondering, is this a normal experience? If not, how do you prevent this from happening? I've tried planning things in advance, but I can't seem to really foresee everything until I start hammering out some code.

Answer 1

This is a very common experience

Most people I interact with, and I myself as well, feel like this. From what I can tell one reason for this is that you learn more about the domain and the tools you use as you write your code, which leads you to recognize many opportunities for improvement after you've already written your program.

The other reason is that you might have an idea in your head about the ideal clean code solution and then the real world and its messy limitations get in your way, forcing you to write imperfect work-arounds and hacks that may leave you dissatisfied.

"Everyone has a plan until they get punched in the face."

What to do about it

To some degree you will have to learn to accept that your code will never be perfect. One thing that helped me with that is the mindset of "If I hate the code I wrote a month ago, it means I have learned and become a better programmer".

A way to alleviate the issue is to constantly be on the lookout for potential improvements as you work and refactor continuously. Make sure to hit a good balance between refactoring and adding new features / fixing bugs. This won't help with big design issues, but it will generally leave you with a more polished code base you can be proud of.

Answer 2

Learn refactoring - the art of gradually improving code. We all learn all the time, so it is very common to realize that the code you have written yourself could be written in a better way.

But you should be able to transform the existing code to apply these improvements without having to start from scratch.

Answer 3

If you have excellent static requirements and understand them well and have time for detailed analysis you have a chance that you can come up with a good design that you'll still be happy with when you're done.

Even in that blissful case, you may learn new language features that would have helped make a better design though.

However, usually, you won't be that lucky: the requirements will be less than stellar and incomplete and although you thought you understood them, it turns out that there are areas that you made invalid assumptions about.

Then, the requirements will change as the users get a look at your initial deliverables. Then something that the users don't control will change e.g. tax law.

All you can do is design, assuming that things will change. Refactor when you can, but realize that time and budget will often mean that your final deliverable is not as elegant as it would have been if you knew at the beginning what you know now.

Over time, you'll get better at digging into the requirements you receive and get a nose for which parts of your design are likely to need flexibility to absorb change.

In the end though, accept that change is constant and ignore the twinge that says "I could have done it better". Be proud and happy that you delivered a solution at all.

Answer 4

This feeling is completely normal, and expected. It means you are learning. Almost every time I begin a new project, I start in one direction, and end up designing it differently in the end.

It's common practice to first develop a prototype before starting on the real thing. It's also common to revisit old code and refactor it. This is easier if your design is modular - you can easily redesign bits and pieces at a time without having to totally trash your old design.

For some people, it helps to write pseudo-code first. Others find it helpful to start with writing comments describing what the code will do, then write the code once the general structure is there. These two things will help you plan your design better, and might avoid the necessity of a rewrite.

If you are part of a team, having a review process is crucial to the design process. Have someone review your code so you can learn how to improve your design.

Answer 5

How can I avoid always feeling like if I completely rebuilt my program from scratch I'd do it much better?

What you could do is to create a throwaway prototype, before start doing the "real" project. Quick and dirty. Then, when you get a prototype to prove a concept, you get to know the system, and how to do things properly.

But do not be surprised, if after N years you come back to this code, and think "what a mess".

Answer 6

Remember this mantra:

The perfect is the enemy of the good.

The perfect solution is not always the ideal solution. The ideal solution is the one which achieves the status "good enough" with the least amount of work.

• Does it fulfill all the requirements regarding functionality and performance?
• Is it free of critical bugs you can't fix in the current architecture?
• Estimate how much work you will invest into maintaining this application in the future. Would the effort for rewriting be more than the long-term effort it would save?
• Is there a possibility that your new design might actually make things worse?

If you answer yes to all of these questions, then your software is "good enough" and there is no good reason to rewrite it from scratch. Apply the design lessons you've learned to your next project instead.

It is perfectly normal for every programmer to have a couple of messy programs in their past. It happened several times over my career as a software developer that I looked at some code, wondered "What idiot wrote this mess?", checked the version history, and noticed that it was me from several years ago.

Answer 7

I've tried planning things in advance, but I can't seem to really foresee everything until I start hammering out some code.

It is tempting to think that perfect planning will give you perfect software design/architecure, however, it turns out that's categorically false. There's two big problems with this. Firstly, "on paper" and "the code" rarely match, and the reason is because it's easy to say how it should be done as opposed to actually doing it. Secondly, unforeseen changes in requirements become apparent late in the development process that couldn't have been reasoned about from the onset.

Have you heard of the Agile movement? It's a way of thinking where we value "reacting to change" as opposed to "following a plan" (among other things). Here's the manifesto (it's a quick read). You can also read up about Big Design Up Front (BDUF) and what the pitfalls are.

Unfortunately, the corporate version of "Agile" is a bunch of bogus (certified scrum masters, heavy process in the name of "Agile", forcing scrum, forcing 100% code coverage, etc), and usually results in asinine process changes because managers think Agile is a process and a silver bullet (of which it is neither). Read the agile manifesto, listen to people who started this movement like Uncle Bob and Martin Fowler, and don't get sucked into the nonsense version of "corporate Agile".

In particular, you can usually get away with just doing TDD (Test Driven Development) on scientific code, and there's a good chance your software project will turn out pretty darn well. This is because successful scientific code mostly has ultra-usable interfaces, with performance as a secondary (and sometimes competing) concern, and so you can get away with a more "greedy" design. TDD kind of forces your software to be ultra-usable, because you write how you want things to be called (ideally) before you actually implement them. It also forces small functions with small interfaces that can quickly be called in a simple, "input"/"output" fashion, and it puts you in a good position to refactor in case requirements change.

I think we can all agree that numpy is successful scientific computing software. Their interfaces are small, super usable, and everything plays nicely together. Note that numpy's reference guide explicitly recommends TDD: https://docs.scipy.org/doc/numpy-1.15.1/reference/testing.html. I've used TDD in the past for SAR (Synthetic Aperature Radar) imaging software: and I can also assert that it works extremely well for that particular domain.

Caveat: The design part of TDD works less well in systems where a fundamental refactoring (like deciding that you need your software to be highly concurrent) would be hard, like in a distributed system. For instance, if you had to design something like Facebook where you have millions of concurrent users, doing TDD (to drive your design) would be a mistake (still okay to use after you have a preliminary design, and just do "test first development"). It's important to think about the resources and the structure of your application before jumping into the code. TDD will never lead you to a highly available, distributed system.

How can I avoid always feeling like if I completely rebuilt my program from scratch I'd do it much better?

Given the above, it should be somewhat evident that a perfect design is actually impossible to achieve, so chasing a perfect design is a fools game. You can really only get close. Even if you think you can redesign from scratch, there are probably still hidden requirements that haven't shown themselves. Furthermore, rewrites take at least as long as it took to develop the original code. It almost certainly won't be shorter, since it's likely that the new design will have unforseen problems of its own, plus you have to re-implement all the features of the old system.

Another thing to consider is that your design only really matters when requirements change. It doesn't matter how bad the design is if nothing ever changes (assuming it's fully functional for the current use cases). I worked on a baseline that had a 22,000 line switch statement (the function was even longer). Was it terrible design? Heck yea, it was awful. Did we fix it? No. It worked just fine as it was, and that part of the system never really caused crashes or bugs. It only got touched once in the two years I was on the project, and someone, you guessed it, inserted another case into the switch. But it's not worth taking the time to fix something that is touched so infrequently, it just isn't. Let the imperfect design be as it is, and if it aint broke (or constantly breaking) then don't fix it. So maybe you could do better...but would it be worth a rewrite? What will you gain?

HTH.

Answer 8

I fully agree with the answer provided by Andreas Kammerloher yet I'm surprised no-one has yet suggested learning and applying some coding best practices. Of course this is no silver bullet, but using open-oriented approach, design patterns, understanding when your code smells and so on will make you a better programmer. Investigate what is the best use of libraries, frameworks etc. There's a lot more for sure, I'm just scratching the surface.

It doesn't mean you won't look at your old code as a total rubbish (you'll actually see the oldest programs even more rubbish than you do without this knowledge) but with every new piece of software you write you'll see you improve. Note also that the number of coding best practices increases over time, some simply change so you'll actually never get to perfection. Accept this or quite the path.

One more good thing is having a code revision. When you work alone it's easy to cut corners. If you have a second person reviewing your code, they'll be able to point out where you don't follow those best practices. This way you'll produce better code and you'll learn something.

Answer 9

To add to the other excellent answers here, one thing I find helpful is knowing where you want to get to.

It's rare to be given the go-ahead for a major bit of refactoring on its own.  But you can often do smaller bits of refactoring as you go along, ‘under the radar’, as you work on each area of the codebase.  And if you have a goal in mind, you can take these opportunities to move, step by step, in the right direction.

It might take a long time, but most of the steps will improve the code, and the final result will be worth it.

Also, feeling like you could do better is a good sign!  It shows that you care about the quality of your work, and that you're evaluating it critically; so you're probably learning and improving.  Don't let these things worry you — but don't stop doing them!

Answer 10

You have inadvertently stumbled upon one of mankind’s greatest challenges (adventures), the bridge between man and machine. The bridge between man and physical structure, Civil Engineering for example, has been in progress for about 200 years or more.

Since software development really only became mainstream in the 90’s, it is approximately 30 years old. We’ve learned that it is not so much an engineering discipline as a social science, and we have only just begun.

Yes, you will try TDD, Refactoring, Functional Programming, the Repository Pattern, Event Sourcing, MV something, Java Script (<- Do this, it’s crazy), Model Binding, No Sql, Containers, Agile, SQL (<- do this it’s powerful).

There is no one fix. Even the experts are still grasping at straws.

Welcome, and be warned, it’s a lonely place; but absolutely fascinating.

Answer 11

I'm going to go a little against the grain. This is incredibly common, but it's not acceptable. What this indicates is that you're not recognizing good ways of organizing your code as you write it. The feeling comes from your code not being straightforward.

Your experience was mine for a long time as well, but recently (the past couple years), I've been producing more code that does not make me feel like I need to throw everything away. Here's roughly what I did:

1. Be clear about what assumptions any particular block of code is making. Throw errors if they aren't met. Think about this in isolation, without depending on details about what the rest of the software is doing. (What the rest of the software is doing influences what assumptions you enforce and what use cases you support, but it doesn't influence whether you throw an error when an assumption is violated.)
2. Stop believing following rules and patterns and practices will produce good code. Throw away mindsets and practices that aren't obvious and straightforward. This one was huge. OO and TDD are usually taught in a way that isn't grounded in practical considerations, as a set of abstract principles you should follow when writing code. But this is completely unhelpful for actually developing good code. If you use OO or TDD at all, it should be used as solutions to problems you understand you have. In other words, they should only be used when you look at a problem and think, "Okay, this makes complete sense and is extremely obvious as a good solution." Not before.
3. When I'm writing code, focus on two questions:
   • Am I using features and libraries in the way they were designed to be used? This includes using it for the kinds of problems it was intended to solve, or at least very similar ones.
   • Is it simple? Will my coworkers and myself be able to follow the logic easily later on? Are there things that aren't immediately obvious from the code?

My code is now more "procedural," by which I mean it's organized by what actions it takes rather than what data structures it uses. I do use objects in languages where standalone functions cannot be replaced on the fly (C# and Java can't replace functions on the fly, Python can). I have a tendency to create more utility functions now, that just shove some annoying boilerplate out of the way so I can actually read the logic of my code. (E.g., when I needed to process all the combinations of items in a list, I shoved the index looping out to an extension method that returns Tuples so that the original function wouldn't be cluttered with those implementation details.) I pass a lot more things as parameters to functions now, instead of having a function reach out to some other object to fetch it. (The caller fetches or creates it instead and passes it in.) I now leave more comments that explain things that aren't obvious just from looking at the code, which makes following the logic of a method easier. I only write tests in limited cases where I'm concerned about the logic of something I just made, and I avoid using mocks. (I do more input/output testing on isolated pieces of logic.) The result is code that isn't perfect, but that actually seems okay, even 2 or 3 years later. It's code that responds fairly well to change; minor things can be added or removed or changed around without the entire system falling apart.

To some degree, you have to go through a period where things are a mess so that you have some experience to go off of. But if things are still so messed up that you want to throw it all away and start over, something is wrong; you're not learning.

Answer 12

By remembering that your time is limited. And your future time is limited as well. Whether for work or school or personal projects, when it comes to working code, you have to ask yourself "is rewriting this the best use of my limited and valuable time?". Or maybe, "is this the most responsible use of my limited time"?

Sometimes the answer will be unequivocally yes. Usually not. Sometimes it'll be on the fence, and you'll have to use your discretion. Sometimes it's a good use of your time simply because of the things you'll learn by doing it.

I have a lot of projects, both work and personal, that would benefit from a port/rewrite. I also have other things to do.

Answer 13

It is a completely normal part of learning. You realize mistakes as you go.

That is how you get better and it is not something you should want to avoid.

Answer 14

You can give yourself the experience to know that the seductive urge to rewrite is usually unproductive. Find some old, hairy, inelegant open source project of moderate complexity. Try to rewrite it from scratch and see how you do.

• Did your from-scratch design end up as elegant as you hoped it would?
• Is your solution really to the same exact problem? What features did you leave out? What edge cases did you miss?
• What hard-earned lessons in the original project did you erase in the pursuit of elegance?
• After you add those missing pieces back in to your design, is it as clean as it was without them?

Eventually, your instinct will shift from thinking "I could rewrite this system so much better" to thinking "this system's cruftiness may be indicative of some complexity that isn't immediately apparent."