Overcoming slow problem solving due to increased knowledge of what might go wrong

Question

This has been troubling me for some time, and I'd really appreciate the input of other professionals.

Short background: I started programming when my parents bought me my first computer in 1988 (at age 14, I'm 39 now). I followed a couple of other career paths before finally becoming a professional programmer in 1997. Late bloomer, perhaps, but that's how it was. I'm still happy with my choice, I love programming, and I consider myself good at what I do.

Lately, I've been noticing that the more experience I gain, the longer it takes me to complete projects, or certain tasks in a project. I'm not going senile yet. It's just that I've seen so many different ways in which things can go wrong. And the potential pitfalls and gotchas that I know about and remember are just getting more and more.

Trivial example: it used to be just "okay, write a file here". Now I'm worrying about permissions, locking, concurrency, atomic operations, indirection/frameworks, different file systems, number of files in a directory, predictable temp file names, the quality of randomness in my PRNG, power shortages in the middle of any operation, an understandable API for what I'm doing, proper documentation, etc etc etc.

In short, the problems have long since moved from "how do I do this" to "what's the best/safest way of doing it".

The upshot is that it takes me longer to finish a project than a novice. My version may be rock solid, and as impenetrable as I know how to make it, but it takes longer.

The "create file" example above was just that, an example. Real tasks are obviously more complex, but less suited for a generic question like this one. I hope you understand where I'm going with this. I have no problem coming up with efficient algorithms, I love math, I enjoy complex subjects, I have no difficulties with concentration. I think I do have a problem with experience, and consequently with a fear of errors (intrinsic or extrinsic).

I spend almost two hours a day reading up on new developments, new techniques, languages, platforms, security vulnerabilities, and so on. The conundrum is that the more knowledge I gain, the slower I am in completing projects.

How do you deal with this?

Answer 1

You're not any slower in completing projects. Previously, you thought your novice projects were done when they really were not. You should sell this quality to clients.

"This company might get it done faster and cheaper, but is it really done? Or will you be hunting bugs for years?"

Beyond that, you need to know and accept the old idiom: "Perfect is the enemy of good."

Answer 2

Sounds like it's time for you to join the dark side: management.

I'm not suggesting you give up programming and become a manager. But it seems like all the experience you've quoted up until now has been technical in nature. In simple operation of writing out a file, you can think of 10 different aspects that a less mature developer would never consider. Not necessarily a bad thing, but...

The dark side is all about present value. It's about making minimal investment for maximum gain (cost-benefit analysis). In business everything boils down to how much will this cost me, probability of success, probability of failure, probability of spectacular failure and potential gain. Do the math; act accordingly.

This works just as well when you are a developer: create a temporary file, ignoring permissions and name collisions - 5 min. Net gain, the rest of the team can start working on whatever code depends on the presence of that file. Is it a perfect solution? Absolutely not. Will it get you 99%, 95%, maybe 90%? Yeah, it probably will.

Another question to ask: How do you feel about technical debt? Some people think it must be eliminated. I think those people are wrong. Just like in business, technical debt allows you to borrow "cash" or "time" in order to deliver something sooner. What's better: A perfect solution in 2 years or a complete hack that a customer can use and purchase in 4 months? Every situation is different, but in some situations if you wait 2 years, your customer will already sign up with your competition.

The key is to manage technical debt the same way a well-run business manages financial debt. If you don't take on enough, you are not getting optimum return on investment. If you take on too much, the interest will kill you.

So my advice: Start evaluating your work based upon whether you're maximizing your value instead of maximizing your thoroughness. And if you practice this, you will develop the same intuition that you already developed in your technical area.

As a side note, I recently started doing the pomodoro technique and it has helped a lot. Instead of going on a tangent of a tangent, focus in small time intervals and then allocate pomodoros for future work/research. It's amazing how many times I made a note to research a topic but an hour later when the time came, I thought to myself, "there's at least 3 other things I could do with my time today which are all more valuable."

Answer 3

I had the (likely) same problem many years ago, it lasted for a few years and I overcame it. So maybe it would be of some interest to you to know how I achieved that, even if I'm not sure my way will also apply to you.

You should also have a look here : The Seven Stages of Expertise in Software Engineering It shows that productivity is in great part a side effect of skill level. It may be that you are still at some point between stage 3 and stage 4 on the technology you're currently using (skill proficiency depends on technology, you can be master of some technologies while still learning others).

Now I start with biographic testimony.

A bit of context. I'm 47. I started programming at 12 in the 80's. While in high school I also worked as a part time professional game programmer. Basically it didn't got me that much money, just enough to buy hardware, but I enjoyed it and learned much. At 18 I started a formal learning of Computer Science.

As a result when I turned 20, whenever starting any programming task I knew many ways to solve the given problems and was very conscious of the many parameters and pitfalls at hands, and drawbacks and limits of any method.

At some points (say about 26 years old) it became really difficult for me to write any program at all. There was so many possibilities opened that I was not able any more to choose between them. For a few years (make it 6) I even stopped programming and became a technical news-writer.

I never totally stopped trying to program nevertheless. And at some point it came back. The key for me was extreme Programming, more specifically the Simplicity principle: "Write The Simplest Thing That Could Possibly Work".

Basically I forced myself not to care about code efficiency (that was my main roadblock, avoid inefficient designs), but just go the easiest way. I also forced me to care less about errors, and delay error handling to a later time, after writing tests raising the error (really that's TDD).

That's something I learned when I was writing. When I do not know what to write, or I knew what I was writing was bad. Just go on. Actually write the bad stuff. I will eventually correct it later. Or if it's really that bad erase it and rewrite it, but it's faster to write things two times that write anything perfect the first time.

Really it is very likely that a code you believe is good at first writing will need as much improvement as a really bad one.

If you follow the Simplicity path you also get an added bonus. You easily accept to remove/change initial design/coding. You get a more flexible mind.

I also got in the habit to put a temporary comment in code, explaining what I was not doing now and intended to do later when the code would be functional in the normal use case.

I also attended an XP Dojo an practiced code katas with other programmers to internalize the XP practices. It helped. It made the above formal methods instinctive. Pair programming also helped. Working with young programmers gives some momentum, but with experience you also see what they do not. For instance in my case I often see them engage in overly complicated designs and I know the design nightmare that may lead to. Gone that way. Did that. Had problems.

The paramount point for me is keeping the flow. Being fast is really succeeding in keeping the flow.

Now I'm back as a professional programmer and I believe I'm both better and faster with a deeper understanding of what I'm doing. Practicing TDD I may still be slightly slower than when I was a young bull (and tested nothing), but I also have no fear refactoring and certainly spend much less time debugging (nearly no time at all, make it less than 10% of time).

Summarily: I overcame my codeblock using agile methods (XP), going for simplicity then refactoring, and practicing to make that instinctive. It worked for me. Not sure it can be generalized to anyone else.

In terms of skill acquisition level, I mostly learned to accept that every time I change technology (learn new language, new framework, etc.) I will go through a stage when I'm slowing down. This is normal and will eventually overcome that. I also can compensate for that with good methodology and general purpose programming skills and It won't be as much a problem.

Answer 4

An important part of programming is managing and controlling complexity and for me personally, it is one of the top issues. If ignored then either the frequency of deficiencies surges or, as in your case, the ETA of finished software increases dramatically.

Software complexity can be controlled and managed from many different levels and ways but a good rule of thumb to get some perspective is this: "Top priority of any software project is customer satisfaction which is a function of their expectations."

In other words, software complexity depends a great deal on how skilled you are at controlling the expectations of your customer.

When one adopts that view, then two important points become obvious:

1. customer expectations must be made explicit (in whatever form);
2. customer expectations can always be modified and that is done through the art of negotiation.

Your example is a very good one, "simply write it" vs "myriad of other considerations". Think about it - if someone were to write down exhaustive requirements for both variants, can there be an equivalence in described features or not?

Building an F16 is different from building a Cessna although both of them can fly.

Answer 5

The simple answer is: accept it.

In all systems there's trade-offs to be made between reliability, robustness, security, speed, hardware cost, development cost, time to market, you name it. You will not always agree with how the customer makes those trade-offs, but you're not the one making those decisions. All you can do is provide a considered opinion. Software development covers the full gamut, from "my personal web page" to "run the nuclear reactor", and the code has to be written accordingly. If a crash means "reload my personal web page" then who really cares if that happens? But if a crash means "Chernobyl" then your preference for solid code is if anything a bit casual for my liking.

There are some clients who will happily accept "Proof of Concept" level code and run it in their live systems, and often they have systems administrators who are well used to that. IME their systems are usually unavailable for an hour or so in the middle of the night while a bunch of scheduled restarts happen. But they've made the business decision that that is how they want to roll. Ideally because their field is so fast-moving that high-quality code would never be ready, but often because they can't see the value (if a system "just works" they never notice it, therefore that system does not represent value for money). If it bothers you too much, try to avoid those clients.

Keeping up to date is time we all have to spend, or at least should spend. Sometimes that will get you work, other times it just keeps your mind in good shape. Either way, try to enjoy it.

Answer 6

It sounds like your skills would be very useful for very high quality mission critical systems development, like finance/trading related applications, broadcasting, aerospace, defense...

Errors in these sort of applications are very costly and they employ people who think like you as you can cover all the cases.

Answer 7

The truth is that modern systems are becoming increasingly complex. The computer is now similar to that game "Jenga" where you have all of these pieces relying on many of the others. Pull out the wrong piece and you have an error, pull out a correct/necessary piece and you still may produce an error. The more complex the system the more time you are likely to spend thinking of ways to make your code more robust, and hopefully more secure as well. Speed would be nice, but I think speed takes a back seat a lot these days when you hear in the news that "XYZ" company was hacked into and 6 million customer's credit card numbers were stolen.

Your clients may not want to hear that their program needs to be secure, robust, etc.. But you could tell them that their car does not need seatbelts and airbags nor their house need locks and doors... because who wants to hear all that?

If you are over-thinking you are going about things the right way, except that you need to pick a strategy that seems "concrete" and just go with it.

Answer 8

It sounds like you're aware of your tendency to think about everything that can go wrong.

Experienced Cautious Developers often learn to follow the mantra YAGNI, you ain't gonna need it, when they try to return to a lean, agile and productive workflow after getting too choked up in the weeds of failure-mode-analysis-gone-amok.

However, if you are indeed writing something in a domain where that level of care is no less than what Professionalism demands, then you should realize that your "velocity", your "productivity", in net terms, is measurable by how much good (or harm) you are doing to your company, your customers, and the software suite, or product family you are building or maintaining.

Remember to:

• Include total cost of maintenance, total cost of ownership, and total cost of deploying and maintaining solutions when you consider changes in your approach. Going faster and making more mistakes may or may not make things better.

• If you work in a good company, you can probably discuss this in your own team, and with your own supervisor, without it being a Career Limiting Move. If you can't, now is a good time to find that out, and find a new job.

Answer 9

Only thing I can see is: "You are becoming more and more valuable".

As and when you get more experience you learn about things you should avoid, and this is what make you better than others.

One thing you would have noticed that your code would be safer and more maintainable now.

• Only thing you need to do is to explain your client why it took time and how it would be useful for them.
• You need to show them depth of your knowledge.
• You need to tell them why you did, what you did and how it would matter them and their business.

My suggestion would be to concentrate on this part.

Answer 10

when in doubt default to badly quoting Knuth...

"Premature optimization is the root of all evil."

Here is what I would suggest, as it seems like you have a problem that I have from time to time...

What really works for me...

1. Write the unit tests, as if all the code was done.
   
2. document the interface.
   
3. implement the interface.

what you have really done:

1. work through the model layers requirements
2. really set up the division of work, which objects are responsible for what
3. get to work in an environment when you can actually step through the working code, which makes things so much faster an more accurate...

also rely on asserts in early development... then figure out which remedies need to be implemented and you wont write code that is unreachable, or hard to test.

Answer 11

I think you should stick to your coding standards, but make sure you are up-front with your clients. Many clients do not know what it takes/costs to build good software. It's part of the professional developer's job to educate them.

Whether you're agile or waterfall, you get some sort of agreement from the client about what they expect the application to do. Too many developers (OK maybe more salespeople) are guilty of sandbagging. "They didn't say they wanted a highly secured website." In most cases, it's because they weren't asked. "Do you mind if your ecommerce site gets hacked?" Of course they care and why would you build it to let anyone penetrate the security? You have to educate them.

Just make sure you're doing only what the client is paying you to do. Part of your service is your experience. Clients expect you to know the pitfalls without them having to ask. It's up to them to include the requirement. You may want to pass on clients that want something cheap.

Answer 12

Think about the practical consequences of a bug as compared to all the other problems that need solving.

Consider the following consequences of creating a poorly written piece of code:

1. Entire database gets dumped every other month. 48 hours of downtime while the backups are restored.
2. Customer records get cross-linked. $200 worth of orders get shipped to the wrong customers per month.
3. A order gets stuck in a wrong status once a week. Order ships but warehouse to has to call helpdesk every time it happens.
4. Once very two weeks or so, the app crashes and user has to re-enter 2 minutes worth of data.
5. Once a month, the app hangs on startup. User has to kill process and start over.

The first one is obviously unacceptable. #2 - #5 may or may not be, depending on the nature of the business. #2 - #5 need to evaluated in the context of other problems the business is facing.

Ideally, #2 - #5 would never, ever happen. In real life, with conflicting priorities, the people signing your paycheck might want you to be working on other things rather than writing perfect code that never, ever has a problem. They aren't going to be impressed if #5 gets fixed at the expense of not fixing a more serious bug in another program.

Answer 13

The solution is to create a collection of libraries with commonly used functions which you can re-use across projects. E.g. I have a StringFunctions.dll .NET library which does stuff like encoding, encryption, decryption, regular expression evaluation, etc. This way, I don't have to continually rewrite things that don't change.

Having a wrapper for file creation tasks also makes lots of sense. Your library could expose a method called GetFile() which does all the checks for you and returns either null or a file (or whatever you deem useful).

Answer 14

@Zilk, I am not great programmer and I am been programming since 1998. Even I am facing this issue now. But what I realized is ultimately quality matters. If I die today, somebody should be able to pickup what I am doing now from where I have left. Such should be the standards of programming (Universal).

I have moved myself from developer to architect now. Moving to Management is changing the line. If you want to continue with your passion you can move to become Architect.

Initially as Technical Architect-->Solution Architect-->Enterprise Architect-->Chief Architect and so on.

As an Architect you will be guiding people to success. People like you who have been programming for decades those years of experience you can utilize to guide others.

Like a bird higher it flies more land it can see so is your experience.

Let me also tell you programming correct implementation is important than programming a wrong implementation faster. Recently one of my juniors programmed something wrong and it cost a bank lots of money. Ofcourse we had delivered on time earlier but that was no use! Was I given the role to guide even though the same junior would have coded that problem would not have happened. I am giving this example to stress that giving good guidance is also important. Some call this job as Consultancy.

Answer 15

I think you need to learn to decide how much needs to be done for which project. Some project may be trivial and you really don't need to spend all that time in perfecting it. Not everything is gonna need rock-solid encryption nor everything will be scaling to million users.

Writing a program which can scale well for more than a million users will take time and experience which you have now, but if you know that your application is not going to be used by more than 1000 users max, there is no point in spending all that time perfecting it.

I think this is an important stage in your programming career and now you need to goto next level, which has to do with maturity and not programming. You need to be able to correctly decide on how much time and code should be enough for any particular project.

And like everything else, you can attain this as well with practice. So try to decide this before you start a project, sometime even while you are already working on it, with experience you will get past that as well. There may be a few hits and misses in the beginning but with time you will perfect it (decision-making, not code).

Answer 16

Another option is: stop writing code, instead sell your expertise in spotting the problems in advance.

In other words, become a Consultant.

Many organizations are happy to pay expensive dollars (if not top-dollar) for someone to spot the issues before spending months on creating the code that makes the problems. It is well known that fixing a bug in design, is orders of magnitude cheaper/easier than fixing it after it has been coded, tested, and deployed.

You won't be writing as much code, and you may likely miss that, but then it seems that the actual lines of code are not your core strength, but in knowing which lines of code should be written - and which shouldn't.

Focus on your strengths.
(well, if that's what you enjoy...)

Answer 17

My best recommendation for you is: building blocks.

Make a file building block that you can trust always, make one for your API, stop wasting your time writing the same thing over and over. Think about every problem once and fix it once and for all.

Noone will catch up to that, certainly not the novice who spend 80% of their time debugging code that fails for corner cases they don't understand.

Most of all, don't fix problems that cannot happen, such as wrong permissions.

If the permissions are wrong, something is already wrong and you should fix that rather than making your program bullet proof.

At some point you have to just not shoot yourself in the foot instead of always checking whether you did or not.

Instead of spending time on documentation, spend time making your code self-documenting and as short as possible. Replace all those duplicate-ish functions. Shrink your library, rename things with precision.

Answer 18

Don't be too hard on yourself. You are working in a profession of increasing complexity, that requires more human intelligence, knowledge and experience than ever before.

Computer processing power is slowing - perhaps soon stalling - leading to the need to introduce multi-core chips, gpu powered numerics, parallelism, etc. There are only so many transistors that can be placed on a chip.

Therefore the big advances presently and in the future are going to come from programmers - advanced algorithms and more efficient code.

If you look at GTA 4 and GTA 5 the differences are astounding. But they both run on the same hardware. This is the result of some very intelligent and advanced programming practice that simply was not required, or available, 10 years ago.

It could also mean that experienced programmers may become more valuable in the future - much like other professions such as engineering where peak earnings typically occur late in the career.

Answer 19

Just like you, I started programming at the age of 14, also when I got my first computer (although I had been studying for a few months at that point). However, I am "only" 33 now. :-)

My suggestion is that, when developing something, you take each one of those worries (file permissions, number of files in a directory, etc.) and then use all of your vast experience to answer a few questions about it, in this spirit:

• How long would it take to handle that issue properly in your code?
• If you don't handle it properly, how likely it is that this thing will bite you later?
• If it does bite you, what are the consequences?

Armed with those answers, such an experienced guy will not have problems to make a wise decision. ;-)

It is the responsibility of "veterans" like you to come up with this type of requirement, and that involves both identifying what can go wrong and deciding which potential problems you should give attention to.

Answer 20

Knowing all the possible criteria's while developing app is the most important thing in developing a quality product. You are doing good in this. One thing you can do is, You can categorize the requirement into quality level then map the level with the given deadline. In this way to you can meet the project deadline easliy.

Answer 21

In the words of Edsger Dijsktra: “If debugging is the process of removing software bugs, then programming must be the process of putting them in. ” You are just doing less of the latter so you have to do less of the former. It's just a question of learning to spend time coding it right. I am still a relatively young programmer (read 20ish) and I aspire to be able to code something completely right once. An hour of planning and 10 minutes of coding is way better than an 10 minutes of planning, an hour of coding, and three of debugging.