I’ve become obsessed with scalability of late and appreciate how open people out there are when it comes to sharing their challenges and solutions.  I talk a lot about the advantages of sharding to scale so it was refreshing to see that they’re thinking the same thing at Digg:

“Ultimately what you want to do is go to a sharded architecture,” said Gorodetzky. “The first 1,000 users go on this set of servers, the next thousand on this set. You’ll find that’s an appropriate way to go, but right now we haven’t done that yet.

It’s also nice to know that they’re at 26 million unique visitors per month without it.

Here is a summary of the tips provided by Digg’s Sys Admin Ron Gorodetzky:

1. Employ master/slave database replication
2. Employ caching to relieve pressure on the database.
3. Scale out your web servers.
4. For horizontal scalability use MogileFS rather than NFS.
5. Experiment with new technologies - They’re experimenting with Puppet for server allocation.
6. Digg uses common tools such as Nagios, and a souped-up version of the Multi-Router Traffic Grapher.

Read the full post here: Digging His Way to Web Success

Related Links:

• Flickr’s Architecture - http://highscalability.com/flickr-architecture
• LiveJournal’s Architecture - http://highscalability.com/livejournal-architecture
• Twitter’s Architecture - http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster
• Amazon’s Architecture - http://highscalability.com/amazon-architecture
• YouTube’s Architecture - http://highscalability.com/youtube-architecture
• Highscalability.com shard posts - http://highscalability.com/tags/shard

A decent article on scaling up your java applications: Scaling Your Java EE Applications

Here’s a list of tips:

1. Make synchronized blocks as short as possible
2. Reduce lock granularity
3. Avoid lock on static methods
4. Use lock free data structure in Java SE 5.0
5. Be ware of race conditions: HashMap bug / consider using ConcurrentHashMap
6. Use Non-Blocking I/O for high concurrency (Tomcat uses Blocking I/O, Glassfish uses Non-Blocking)
7. Parallelize single-thread tasks if possible by seeking independence in order of operations. JOMP or Parallel Java may be used to simplify this task.
8. To scale up in memory use multiple JVMs and limit the memory allocated to each to about 3GB or less.
9. Database - pay attention to isolation levels. keep stored procedures to a minimum.
10. Be ware of operating system and disk I/O limitations such as have a large number of files in one directory.
11. Synchronous logging