5

We current have a very simple Multi Tenant monolith, with a SQL SERVER backend (Self hosted on EC2 on AWS), and multiple application services talking to one DB behind an Classic AWS ELB. Our database has grown to a point now that we are considering splitting it by tenants per region, due to growth in different regions and latency concerns, plus downtime maintenance window considerations. We also want to keep the same DNS...www.domain.com for example for both regions due to existing links etc...

We are considering now

  1. Use Cloudfront to do geo based routing as well as basic cdn caching in front of the services, as an edge proxy of sorts
  2. Shard sql server into 2 main regions, namely Australia and North America, based on tenant's location.
  3. Application services exists on both regions with seperate db.
  4. Have a shard table maybe in dynamodb global table or s3 somewhere, and because we have less than few hundred tenants, and mostly immutable, can be cached for a long time.

Main grey area, what can I do when the route goes to the wrong Region/Db?

For example, say I have

  1. Tenants 1 to 10 in Australia
  2. Tenants 11 to 20 in USA

Tenant 1 went to USA for a holiday and tried to access www.domain.com, based on geo routing, he will be routed to the USA datacenter, which in the database in USA will not contain his data. I was thinking

  • Based on the login user, I can determine the tenant (in the application service), so if he is logged in to the wrong region, based on looking up the shard table, I will reply a 304 redirect, maybe with a cookie of sorts, so cloudfront maybe, with lambda at edge that can read and do extra redirection logic? Still not clear if that is possible or a good solution.

What is the best practices around that? I reckon it would be a solved problem but I cannot seem to find anything more practical with examples with my bad googling skills, most of them are like theory of shard tables etc...

Any advice would be much appreciated.

Community
  • 1
Joshscorp
  • 101
  • 6
  • can you only determine the tenant after the user logs in? most solutions ive seen have a url per tenant – Ewan Oct 18 '18 at 18:15
  • yeah we need the same url because of links that were sent out to millions of customers – Joshscorp Oct 18 '18 at 22:34
  • @Joshscorp do users need to be logged in to get to the final destination of that URL? – RibaldEddie Oct 19 '18 at 00:58
  • yes it is based on the login user – Joshscorp Oct 19 '18 at 11:00
  • An alternative approach may be to allow the application to query the proper database based on the query. Tenant 1 still uses the USA web server. The USA web server sees their data is in Australia and polls the Aussie db for the data before responding. – bitsoflogic Oct 22 '18 at 17:21
  • we tried that, latency is too horrible cross region...as we usually do more than 1 db call per request...would be faster to go via the web service in the different data center. – Joshscorp Oct 30 '18 at 10:24
  • was wondering why there is no read replica in different regions? – lennon310 Nov 28 '20 at 06:00

1 Answers1

2

Main grey area, what can I do when the route goes to the wrong Region/Db?

Re-rout them to the right DB. Seems simple. But...

Tenant 1 went to USA for a holiday and tried to access www.domain.com, based on geo routing, he will be routed to the USA datacenter, which in the database in USA will not contain his data.

...is just the tip of the iceberg. Lets say you solve that problem. Then...

Tenant 2 went to USA and liked it so much they decided to stay. But now and forever more will find that your service is really laggy.

What you need is a system that will turn a geo-routing failure into a re-rout to home but will migrate the data after a set amount of geo-routing failures. That way the only people you really annoy by not having a truly distributed system are frequent fliers.

At which point you have to ask, "how much do we care about frequent fliers?".

candied_orange
  • 102,279
  • 24
  • 197
  • 315