How do people manage to write and maintain extremely complex and hard to read code?

Question

Reading SQLite source code is IMO mission impossible. Yet it is a usable piece of quite complex software (it's a full-blown embedded database after all) that can be downloaded, compiled and used from others code and it is constantly updated.

How do people manage to write and maintain such extremely complex and hard to read code?

Answer 1

In the specific case of SQLite, the main tool they've chosen to use in development and maintenance is automated testing. They pride themselves on 100% coverage (branch coverage, not statement coverage) in their test suite. According to them, it's one of the best-tested software products in the world. So they know right away when something they've added or changed causes a regression, and can develop pretty fearlessly as a result of that.

http://sqlite.org/testing.html

Pretty staggering numbers--they have around 640 times as many lines of testing code as of production code.

EDIT: This question has been raised from the dead, it seems! And a little more than a year later, that same page reports they have 1177 times as many lines of testing code as production!

Answer 2

There is a big difference between Complex and Complicated. The following link about sums it up. :)

http://codebetter.com/blogs/dru.sellers/archive/2009/09/25/complex-vs-complicated.aspx

On a more personal note, I work on a code base of more than 1 million lines of code. I have been in it from line 1 to its current state. The more farmilar you are (read longer you are in the code) the easier it becomes. I can not tell you what every line does, but I can tell you were to start looking for a given task or bug. It just comes naturally.

The moral of the story is that like any thing in the programming world it takes time. If you expect to look at SQLite and just know and understand it, you are kidding yourself. It is going to take time to figure out how everything works together. The difference between good code and bad code is how long that process takes.

Lastly, some developer do not have the ability to jump into a code base and start figuring it out. Many will feel overwhelmed at sheer size or architecture of the code base. Others will have no problem just jumping into it. It all depends on the developers strengths.

Answer 3

• functionalities evolves over time.
  • new functionalities are driven by new customer requirements.
  • Old code was written in a way that allows future yet-to-be-implemented functionalities to be added without breaking old code.
• domain experts (in this case, database is a well-known domain in Computer Science.)
• community feedback
  • bugs
• steep learning curve was not a problem for the well-prepared and persevere. It may take 6 months, 1 year or even longer to become comfortable, and that's what some people are able to put up with.
• commercial motivations. without monetary support, very few people can invest the time and energy into the steep learning curve.

Answer 4

You do not need to have an intimate understanding of the whole project in order to be able to maintain it. Usually with large, complex software, people will have their own particular "areas" that they look after and they only have a 'passing' knowledge of the rest of the system.

SQLite is actually relatively small on the scale of "large software projects" but if you look at something like the Windows operating system, you'll have people who just work on the kernel, people who just work on the shell, people who just work on Internet Explorer, people who just work on the Window manager, etc etc. Someone who works in the "shell" isn't going to be able to fix a bug in the kernel at the drop of a hat.

There's also the benefit that these projects evolve over time: they didn't always start out this complicated. That means a new developer can usually be "trained" by more experienced developers.

When you join a large team of developers, you'll be given a particular aspect of the project to work on (maybe a bug or new feature) and you'll have another developer be you "buddy" for the first few iterations. Your buddy will have a good understanding of the area you're working and can help you find your way around.

For open source projects like SQLite, it's actually a little harder because there's no motivation for existing developers to "train" new developers. So you'll find you're on your own a bit more. But you can still find help on developer forums or mailing lists (e.g. just posting a question like "I'd like to implement such&such feature" or "I found bug XYZ, where do I start looking?" and you're likely to get some form of help.

Answer 5

There is a difference between hard to read and complex project. If a person doesn't understand deeply the language which the project was written or the domain of the project it doesn't means the code is badly written.

My advice:

• Have sure you understand the language of the project. It's not enough know the language.
• Learn every detail about the domain before put hands on code.

For me SQLite is far to be complex and nothing complicated. The domain of this project demands deep understanding about wide concepts.

Answer 6

One thing not mentioned in the other answers is "It comes naturally to them". Most people want to write good code but they are tuned to writing bad code. Some of the programmers I have met are naturally LINEAR thinkers + some times very wittingly they come up with complex solutions. Most of such people don't spend time reading or learning from the book they learn as and when the work demands it.

Answer 7

How do people manage to write and maintain such extremely complex and hard to read code?

If they are the original coders, and they keep maintaining it, they do not see it as "complex and hard to read". They probably see it as "simple and easy to read", since they wrote it themselves.

Any code takes time to understand it. It's just that some take longer than others :)

Answer 8

You can wrap your head around any code-base if you have - perseverance, patience and a methodical approach - but mainly perseverance :-)

Answer 9

The algorithm being implemented sets a lower limit on how simple the code for an implementation can be. If an abstract description of the algorithm to be implemented is extremely complicated and requires tons of different pieces of data to be coupled to each other, then the code implementing said algorithm can't be simple no matter who's writing it.

In other words, one reason why I sometimes write inscrutable code is because, given the algorithm I want to implement, I have no choice.

Answer 10

• If you're the original author of the software, and you've worked with it for > 10 years, and you're a genius, then it may be possible to understand the whole thing. Big pieces of complex software (like SQLite or the Linux kernel) indeed usually have exactly one original author having a complete, deep understanding of it, even if other people have contributed.
• If the software architecture is sane (high cohesion, low coupling), understanding the whole thing is not a prerequisite for making useful additions and modifications to it.
• Understanding a RDBMS or OS are somewhat special cases, requiring deep understanding of the underlying CS principles. Not so with "ordinary" software applications.

Answer 11

Managing becomes a lot easier if there are things done always the same way. I do not know the SQLite code, but I'm working in an environment where there are multiple projects. Besides understanding the business case (eq logic), as everything is basically done the same way everywhere (database access, etc.) it's relatively easy to start working on another project. Long text short: Coding guidelines - in appropriate manner - make life and such code far more easy. This could be one of the helpers for the SQLite coders.

Answer 12

They try to write it in a simple way but not a simplist way.

Going as much simple as you can makes things faster to understand/read, even if it can take some time.