1

I have different availability requirements (90%, 95%, 99% guaranteed uptime) for certain subsets of data. For example, the subsets of data are split based on the possible values A, B, C.

What would be a good way to do this? Guaranteeing 99% uptime for all sets would be a waste of resources and too expensive given the requirements.

I thought of using replicated clusters, where for each subset, queries go to a certain cluster that has more, or less replicas. But then how is each query routed to the correct cluster?

Perhaps with sharding, where the shard key value is A, B or C?

In both cases I wonder what happens if the (key shard) value is updated? E.g. A -> B. Will the corresponding record be migrated to another shard/cluster? And does this have a negative impact on the performance?

Senne Verhaegen
  • 113
  • 3
  • 3
    I'm struggling with the idea that designing, implementing, and verifying that the data is written to and read from the right cluster is cheaper than using data storage solutions that support local clustering and near-real-time replication (to distributed data centers, ideally). By making sure your data center has approriate security and infrastructure (redundant power and networking, HVAC, fire control) and configuring clustering and replication should get you pretty close to your 95-99% uptime. – Thomas Owens Jul 27 '22 at 10:52
  • I guess you are right. The sheer complexity of the approach would indeed be expensive as well. 99% allows for a down-time of about ~15 minutes / day which seems pretty doable. – Senne Verhaegen Jul 27 '22 at 12:03

1 Answers1

6

Designing, implementing, and verifying application-side controls to manage which shard or cluster contains the data and that reads and writes are happening against the shard is probably going to be more expensive than taking advantage of out-of-the-box clustering and replication solution.

Availability begins with the data center. Using any reputable data center would get you things like physical security, redundant power and networking connectivity, fire control systems, and HVAC. You'd need the redundant power and networking to start to achieve the desired uptime, and the other physical safety controls would help in disaster scenarios.

Once you have the data center under control, the mainstream file and data storage tools offer built-in configurations around sharding and replication. These implementations are probably going to be more robustly designed and tested than any in-system features that you design. For open-source tools, they also benefit from many eyes.

99% availability is really a lot of downtime. It's 00:14:24 per day - fourteen minutes and twenty-four seconds. Annually, it's over 3.5 days. Depending on your SLAs, planned downtime may or may not be considered against that 99% uptime. Regardless, that's a lot of downtime.

It would be probably be cheaper and less error prone to use a data storage solution's out-of-the-box sharding and replication functionality. Going with a managed service from a large provider that undergoes audits of their processes and capabilities, like Microsoft, Amazon, or Google, would increase the confidence that the desired uptime is being delivered.

Just as examples: A multi-availability zone deployment of AWS's RDS service for managed relational databases would get you to 99.5% availability. For files, AWS's S3 has a 99.9% monthly availability and grants 100% service credit if the availability is below 95% for a month. Going with services like these and allowing the developers to focus on your core competencies would probably pay for itself pretty quickly while exceeding your more strict availability requirements.

Thomas Owens
  • 79,623
  • 18
  • 192
  • 283