What best practices should be undertaken for a website that needs to "scale out" to handle capacity? This is especially relevant now that people are considering the cloud, but may be missing out on the fundamentals.
I'm interested in hearing about anything you consider a best practice from development-level tasks, to infrastructure, to management.
Design for Concurrency
That is, as you're coding, plan around having multiple threads going. Plan the shared state (often just the db). Plan for multiple processes. Plan for physical distribution.
This allows you to distribute your system across multiple machines, and across multiple processes with load balancing. It allows you to have redundant processes running in case of failure, and in case you need to modify the system in-place, you don't have to kill all service to do so.
A few things you might consider:
Share Nothing architecture.
With that in mind, and contrary to what you might think, don't jump to a scale-out solution right away. The off-system overhead vs. an in-system call should not be under-weighed. For instance, it takes a LOT longer to make an DB connection across any network interface than it does to make a local call. Budget how much time in management, power, and tuning effort is needed in scale-out vs. the extra $ for a true large system.
Regardless, I there is still great value in "share nothing" architectures and you can layer and scale-out your systems when the time comes.
Secure, Fast, Reliable DNS
I found a few high-capacity websites using the registrar's DNS server, which had no SLA for uptime or performance. In addition, their servers were located in India and the latency alone increases the chance that a DNS spoofer could poison your customer's, or intermediate ISP's cache. This would cause even your SSL-protected traffic to be redirected without anyone knowing.
DNS speed also affects the initial load time of your server, before the records are cached.
I use DynDNS or Neustar to most of my customers since they have a pretty solid DNS infrastructure (though it's expensive and I have no other affiliation to those companies).
Parallelize requests over several hostnames
Part of the HTTP standard is a section that says webclients will request a maximum of 2 sessions per DNS Host. Here is a solution where you and alias out your www.domain.com and get a higher request concurrency, making your page load faster:
Basically it involves editing your ASP.NET HTTP Handler to alternate the target hosts you send clients to, where each host is a CNAME to "www".
I think that the key is going to be simple:
Have simple code. That means something that you look at and understand. As you expand and change servers you need to know what is going on. You also might need to add coders who need to quickly understand. Hooks and XML files that call random code that is not obvious is very bad.
Then you can test and find the problems.
We at stellarbuild try to make sure our websites scale without down time. That means you need to be able to know what your code does and where it does it. Even if you are testing a different machine you can't take too long to scale. Most people only start when it is almost too late, sadly. You can optimize only once you do that in my opinion.
