How do you manage extensibility in your multi-tenant systems?

Question

I've got a few big web based multi-tenant products now, and very soon I can see that there will be a lot of customizations that are tenant specific.

An extra field here or there, maybe an extra page or some extra logic in the middle of a workflow - that sort of thing.

Some of these customizations can be rolled into the core product, and that's great. Some of them are highly specific and would get in everyone else's way.

I have a few ideas in mind for managing this, but none of them seem to scale well. The obvious solution is to introduce a ton of client-level settings, allowing various 'features' to be enabled on per-client basis. The downside with that, of course, is massive complexity and clutter. You could introduce a truly huge number of settings, and over time various types of logic (presentation, business) could get way out of hand. Then there's the problem of client-specific fields, which begs for something cleaner than just adding a bunch of nullable fields to the existing tables.

So what are people doing to manage this? Force.com seems to be the master of extensibility; obviously they've created a platform from the ground up that is super extensible. You can add on to almost anything with their web-based UI. FogBugz did something similiar where they created a robust plugin model that, come to think of it, might have actually been inspired by Force. I know they spent a lot of time and money on it and if I'm not mistaken the intention was to actually use it internally for future product development.

Sounds like the kind of thing I could be tempted to build but probably shouldn't. :)

Is a massive investment in pluggable architecture the only way to go? How are you managing these problems, and what kind of results are you seeing?

EDIT: It does look as though FogBugz handled the problem by building a fairly robust platform and then using that to put together their screens. To extend it you create a DLL containing classes that implement interfaces like ISearchScreenGridColumn, and that becomes a module. I'm sure it was tremendously expensive to build considering that they have a large of devs and they worked on it for months, plus their surface area is perhaps 5% of the size of my application.

Right now I am seriously wondering if Force.com is the right way to handle this. And I am a hard core ASP.Net guy, so this is a strange position to find myself in.

Answer 1

I faced a similar problem, and I'll tell you how I went about resolving it.

1. Firstly, there is a "core" library, or engine. This basically runs the show, much as you have already got figured out. It handles things common to every system, from rendering forms dynamical, user and account management, roles, you name it, it does it.

2. Every part of the system is contained within a module. A module has dependencies (other modules that it depends on). For example, the "core" system has Security (users, groups, roles, password policies), Locale, (translations, countries, cultures), File Repository, Email, etc, etc. Each module defines itself using an xml file. The xml file basically specifies the schemas, tables, calculation classes, screen definitions etceteras. These are read in when the application starts if the file date has changed.

3. Customer-specific modules have their own xml files and own DLL. It all plus in and functions accordingly and transparently with the rest of the system. Right down to replacing existing MVC views with custom ones, with custom code and custom view models.

4. If a customer wishes to extend existing functionality, the xml/system provides a method where I can "derive" one module from another. The new module has all the existing functionality, but with the customer's specific requirements in a new DLL and with an extended XML file that can make the modifications. The Caveat is, they can't actually remove existing fields in this system ,but we can extend it and provide entirely new objects and functionality.

Answer 2

Having lots of versioning logic and layers of code to manage isn't adding value to your web app / site. It also requires more brain power to understand what is going on and distracts you from the core of what you are doing.

I advise differently. I advise to keep things simple over complex.

Maintain a single code base that is the same for everyone. Every feature you add is a feature for all. Only restrict the number of items or storage or some quantifiable thing, not features. The reason being is quantifying things such as storage, number of data entries, etc is so simple to program against and it can be applied to just a small handful of things that really does restrict the user from going buck wild on your app. It only needs to be programmed upon the adding of items instead of doing some feature logic. What if features are tied into other features? It becomes very complicated to code for this.

To qualify my answer, I have built an ecommerce framework that I have for many clients and have learned the hard way of trying to maintain all their idiosyncrasies. I ended up breaking the chain and created a single administration website that is multi-tennant for ecommerce. I then have websites that pull in data from web service calls. This allows different teams in different technologies to implement specific features for ecommerce sites while I keep getting paid each month on the same infrastructure and I don't care what they are doing. I just limit the usage by numbers, not features. Its way easier. The client is able to choose their own vendor to build their website and I'm not involved anymore in those decisions.

I also have a subscription service for content management SAAS. This works by usage tiers as well and the clients can add their own plugins if they want while I have my team add more features for all.

This is just my experience from banging my head into the wall after so long, then stumbling upon something simpler.

If something is always going to be specific for each client, don't make a framework for it. Break it down to its simplest parts so the framework portion works for all and then cut the rest out so you can extend it for the individual.

Answer 3

The software product that I work on has extra fields for user extensibility. Each data item for these fields can be keyed to an existing row in the main tables in the application. For example if your software contains customers and each customer a list of orders, then the customizable data table will have a column for foreign keys to the customer table and the order table, only one of which will be non-null. Either that or a number of tables each one having the custom fields for one table.

This is a simple and performant way to store extra data for the user. Additional information would need to be stored about the names and types of these fields.

That covers the custom fields for a customer. For custom behaviour, a web services API would allow for extensions.

Answer 4

1) Running someone else's code on boxes you own is a quite tough problem. Either you can trust them or you can defer responsibility reasonably, you will have to sandbox their code very heavily and have a higher-than-average dedication to security. As your plugin system allows more functionality, the difficulty of making it secure grows. Things like denial of service (plugins consume too much resources), information leakage (plugins can access other tenants' data), etc. can be very real and troublesome, I wouldn't take part into this unless I could get people very acquainted with these problems or very capable people with lots of time to get it right.

2) Yes, modularity and customizability are tough. Things such as the Strategy pattern can help (a fancy name for function pointers; in Java it's typically implemented by declaring an Interface which users of the class can implement to customize the behaviour of your code), but you'll want very isolated and decoupled code for this to work.

3) Data-wise, this might be one of the examples of schemaless storage being useful. If you have a relational data model, having tables with lots of columns for different customers is problematic (yes, you end up with lots of nullable columns, which is bad; one reason is that it's hard or impossible to set the correct constraints [i.e. constraints which do not allow invalid combinations of nulls to appear]). Adding customer-specific tables (i.e. users and foo_users, with users holding the fields valid for all customers and foo_users having the Foo specific fileds) is better; you can have correct constraints easily, but your RDBMS might not handle it gracefully (due to join explosion, limits on number of tables, etc.) and will probably look ugly in your code.

You might end up implementing a key-value store in your RDBMS, which makes your data model less relational-ish and will cause discomfort (i.e. relational databases work best with relational-ish models). I'm not sure whether a NoSQL solution will fit the problem overall, but the schemaless-ness of most implementations might be an advantage.