0

I know questions like this has been asked before. But none of them truly answered me.

How to keep a big and complex software product maintainable over the years?
How do you organize highly customized software?
...

We're working on a software that has a lot of rules and conditions. Also, those rules change frequently.

So far we've used many designs and techniques to cope with the requirements and implement them. To be more specific, let us focus on the following examples:

  • Using configurable options, so that values can be changed easily by business people, like the amount of debt a customer can undergo.
    Problem: Soon there are like hundreds of configurable options, and even naming them becomes a nightmare, let alone managing them, changing them, tracking them in code, etc.

  • Using small satellite project files as additives to the main engine (something like extensible workflow), so that new rules can be added hot, and also unwanted rules can be removed from the process pipelines.
    Problem: Soon the number of project files become so huge that the file-system management of the projects becomes a pain in the ass. What if for a short period of time business team needs a special customization that only lasts for two months, and then it should be removed from the system. Something like prepaid registration to track user-behavior for making the real decisions. Or a rule as simple as banning customers with suspicious activity that should be launched really soon.

It seems that our team is going the wrong path altogether. So how a team can handle creating highly-complex, highly-changeable software when it encounters the specific problems mentioned? What points should that team consider? Are there specific architectures or designs that will help us creating such software?

Community
  • 1
Saeed Neamati
  • 18,142
  • 23
  • 87
  • 125
  • @DocBrown, but yes yes yes. For a team who repeats himself a lot, the concept of encapsulation and inheritance becomes the answer. Once introduced to the concept, they can efficiently solve their problems. Of course, it was just a simple example. – Saeed Neamati Nov 27 '15 at 20:54
  • @DocBrown, come on, I just said that it's a simple example to make the things clear. If we knew the answer, why would we ask a question as you said? – Saeed Neamati Nov 27 '15 at 21:04
  • 6
    The reason questions like this are never "truly answered" is because they are too broad to be answerable. If this question does have an answer, it'd probably take the form of a several hundred page book, and even then I'd expect to see many equally long books that strongly disagree with each other (if this isn't already the case). Alternatively, [you could join us in chat](https://chat.stackexchange.com/rooms/21/the-whiteboard) and see if a proper two-way conversation where we compare our experiences helps clarify things at all. – Ixrec Nov 27 '15 at 21:29
  • @Ixrec: before closing this question, why not give the OP a chance and let him restrict the focus on the two cases he described well? Those two are very answerable. – Doc Brown Nov 27 '15 at 21:33
  • 1
    @DocBrown its often easier to restrict the focus when the question is closed so that other answers don't come in and answer the "wrong" part of the focus that would need to be changed when the question is edited. Its rather easy to reopen a too broad question (currently) if the scope is sufficiently narrowed. –  Nov 27 '15 at 21:34
  • 3
    @DocBrown The reason I left that big comment is because I'm hoping he does restrict the focus, and when he does, then I'll vote to reopen. Merely closing a question is not meant to be an insult or a punishment. – Ixrec Nov 27 '15 at 21:35
  • 1
    You might embed some command interpreter like [Guile](http://www.gnu.org/software/guile/) or [Lua](http://lua.org/) in your app. See also [this](http://stackoverflow.com/a/33968896/841108) – Basile Starynkevitch Nov 28 '15 at 07:49
  • @Ixrec: I changed the question a little to make it more clear that the "too broad" parts are not in the focus of this question. Please check if you still think it is too broad. – Doc Brown Nov 28 '15 at 10:34
  • I completely disagree. This question is not broad at all. And the answer is not a book. Analogy => Question: "We can't create tall buildings no matter how much stronger wood we use, should we use another material?" Answer: "Yes, use cement, instead of wood". Honestly I can't understand what is not clear that makes it too broad. It's as simple as asking "hey, we're stuck at creating highly complex highly changeable software, what should we do" and the answer might be "are you familiar with X architecture or Y tool, or Z approach? That's what other people are using". – Saeed Neamati Nov 29 '15 at 08:03
  • @BasileStarynkevitch, you truly understood my question. Your answer is a type of answer I'm seeking. Thanks for suggestions. We're looking now. – Saeed Neamati Nov 29 '15 at 08:11
  • @SaaedNeamati: it is like that, except the equivalent of cement hasn't been invented yet. In my view, projects with highly changable, very complex rules _always go way over budget before they fail_. Organisations should learn that we can't really do that with software. – RemcoGerlich Nov 29 '15 at 08:26
  • sounds like you need a DSL or a rule engine the business people could use themselves... – AK_ Nov 29 '15 at 10:11
  • How to present a large number of configuration options in a user-friendly way might also be a question for [User Experience Stackexchange](https://ux.stackexchange.com/) – Philipp Nov 29 '15 at 11:31
  • Ultimately a lot of the complexity comes from the problem itself. You can and should eliminate most of the complexity overhead of the solution with clever techniques but you can't get rid of the complexity inherent in the world you're trying to model. In plain English: maintaining 100 slightly different systems is hard. It needn't be a 100 times harder than maintaining a single one but it will always be considerably harder than just a single one. – biziclop Nov 29 '15 at 20:19

2 Answers2

6

I will focus on the two core problems described in your question:

Soon there are like hundreds of configurable options, and even naming them becomes a nightmare, let alone managing them, changing them, tracking them in code

The naming problem can be solved by organizing them in hierarchies. For example, look at the thousands of options you will find in the about:config page of the Firefox web browser - they are well-structured and organized in groups of multiple levels.

Tracking in code should follow the same principle: create a hierarchy of objects or data structures (actually, naming and code hierarchies should match, and one should be generated from the other). Of course, that alone is not enough, for hundreds of options you will need to maintain some formal documentation in a disciplined manner (again, check how the Firefox community solves this). You need to establish some quality gates like reviews, and you should to clean up things from time to time. In the end, this is a purely organizational problem, but it is in fact in no way different from managing something like a large database schema.

Using small satellite project files as additives ... Soon the number of project files become so huge that the file-system management of the projects becomes a pain in the ass

That is actually not different from the first case: hierarchies, formal documentation, quality assurance and some discipline will help you here, too. Moreover, assumed your additives are code, there should be no real difference to code which is not "hot-pluggable" - the latter makes only a difference for the user of the system, but not for the maintenance by developers. So you have to care for the same things you always have when writing bigger software: dependencies, clean architecture, SOLID code etc.

As you mentioned "a business team": another option to keep "additives" more manageable is to provide an extension mechanism like a script or query language, so your business team can write such additives on their own. That delegates the burden of managing the extensions away from the development team to the users of the system on the "business side". Though that sounds just like shifting the problem around, it can actually help you, because you create a strict boundary between your core system and those extensions, with in total more people involved for doing the management tasks that give you a headache.

And maybe the people "inventing" all these new requirements will be thinking twice what they do, if they have to write these scripts on their own.

One final thing to add: I heavily recommend to keep new options as "orthogonal as possible". In other words: each new option should have as few dependencies as possible to other existing options, ideally none. That typically implies to split up new requirements for "one option" into several smaller options, but I can tell you from experience, that really a key factor to keep such a system manageable.

Doc Brown
  • 199,015
  • 33
  • 367
  • 565
  • 1
    This is a good starting. Thanks for answering and helping. What if FireFox business team today say that "each new tab should ask user if he wants to create it with advanced rendering support or not", and then in two weeks based on users' feedback, they say that "we don't want to ask them that anymore". What if options like that can enter and leave system at 100 options per week? I think about a rule engine. But I can't connect the concept of rule engine with our problems. – Saeed Neamati Nov 29 '15 at 08:08
  • @SaeedNeamati: I imagine the Firefox team would simply add a boolean option for this behaviour and let the users decide if they want the feature or not. Maybe they would change the default setting of such an option with the next release, but that would be pretty simple once the feature is available. The question if a "rule engine" makes sense for your case you will have to decide on your own, if you have a *specific, focussed* question about this, you can try to ask it at Programmers, in a new post. – Doc Brown Nov 29 '15 at 08:44
  • `And maybe the people "inventing" all these new requirements will be thinking twice what they do, if they have to write these scripts on their own.` Even having to specify exactly what they need might make them realise they don't really need it. :) – biziclop Nov 29 '15 at 20:15
1

People who work in software product lines frequently wrestle with the same problems - how to write software that can vary depending on the deployment environment. There are several methods that have been identified to manage variability - build-time configuration, plug-in architecture models, run-time configuration of properties, configuration files, extending or modifying the source code for each variant. However, like you've found, there's no perfect solution. There are just trade-offs between the options.

To answer your question - yes, there are processes and methods, tools, architectures, and built software systems that achieve what you're trying to achieve. However, without a much deeper understanding of your software system and your organization, it's not possible to provide you with guidance on what, exactly to do.

The field of software product lines is very big. It ranges from requirements development through software maintenance and includes organizational structure, project and program management, and more. The books Software Product Lines: Practices and Patterns and Software Product Lines in Action: The Best Industrial Practice in Product Line Engineering can give you a good introduction to product line engineering, which addresses variability and reuse. More focused on software reuse is Software Reuse: Architecture, Process and Organization for Business Success.

Thomas Owens
  • 79,623
  • 18
  • 192
  • 283