How Digg Scales

I’ve become obsessed with scalability of late and appreciate how open people out there are when it comes to sharing their challenges and solutions.  I talk a lot about the advantages of sharding to scale so it was refreshing to see that they’re thinking the same thing at Digg:

“Ultimately what you want to do is go to a sharded architecture,” said Gorodetzky. “The first 1,000 users go on this set of servers, the next thousand on this set. You’ll find that’s an appropriate way to go, but right now we haven’t done that yet.

It’s also nice to know that they’re at 26 million unique visitors per month without it.

Here is a summary of the tips provided by Digg’s Sys Admin Ron Gorodetzky:

  1. Employ master/slave database replication
  2. Employ caching to relieve pressure on the database.
  3. Scale out your web servers.
  4. For horizontal scalability use MogileFS rather than NFS.
  5. Experiment with new technologies - They’re experimenting with Puppet for server allocation.
  6. Digg uses common tools such as Nagios, and a souped-up version of the Multi-Router Traffic Grapher.

Read the full post here: Digging His Way to Web Success

Related Links:

Lions and Tigers and Bears. (and Patterns) Oh My!

I just read a great post that describes and compares the often confusing and scary landscape of patterns for achieving separation of concerns in applications requiring user interaction.