Entries from July 2008 ↓

How Digg Scales

I’ve become obsessed with scalability of late and appreciate how open people out there are when it comes to sharing their challenges and solutions.  I talk a lot about the advantages of sharding to scale so it was refreshing to see that they’re thinking the same thing at Digg:

“Ultimately what you want to do is go to a sharded architecture,” said Gorodetzky. “The first 1,000 users go on this set of servers, the next thousand on this set. You’ll find that’s an appropriate way to go, but right now we haven’t done that yet.

It’s also nice to know that they’re at 26 million unique visitors per month without it.

Here is a summary of the tips provided by Digg’s Sys Admin Ron Gorodetzky:

  1. Employ master/slave database replication
  2. Employ caching to relieve pressure on the database.
  3. Scale out your web servers.
  4. For horizontal scalability use MogileFS rather than NFS.
  5. Experiment with new technologies - They’re experimenting with Puppet for server allocation.
  6. Digg uses common tools such as Nagios, and a souped-up version of the Multi-Router Traffic Grapher.

Read the full post here: Digging His Way to Web Success

Related Links:

Cross-language development.

I started looking at Erlang about a year ago and would have used it if it wasn’t so difficult to develop cross-language solutions. I wanted to be able to use the guts of Erlang for concurrency, but re-use existing algorithms written in Java and C# within each Erlang process. It looks like Facebook engineers may have done just that with Thrift, a cross-language serialization and RPC framework.

One of the downsides of multi-language development is reusing code across languages. Facebook created Thrift to tie together the Babeling Tower of all their different implementation languages. Thrift is a software framework for scalable cross-language services development. It combines a powerful software stack with a code generation engine to build services that work efficiently and seamlessly between C , Java, Python, PHP, and Ruby. Another approach might be to cross language barriers using REST based services.

This is very cool! I love the idea of being able to marry the strengths of multiple languages. I’ll be taking a closer look at this over the coming months.

Scaling Up Your Java EE Applications

A decent article on scaling up your java applications: Scaling Your Java EE Applications

Here’s a list of tips:

  1. Make synchronized blocks as short as possible
  2. Reduce lock granularity
  3. Avoid lock on static methods
  4. Use lock free data structure in Java SE 5.0
  5. Be ware of race conditions: HashMap bug / consider using ConcurrentHashMap
  6. Use Non-Blocking I/O for high concurrency (Tomcat uses Blocking I/O, Glassfish uses Non-Blocking)
  7. Parallelize single-thread tasks if possible by seeking independence in order of operations. JOMP or Parallel Java may be used to simplify this task.
  8. To scale up in memory use multiple JVMs and limit the memory allocated to each to about 3GB or less.
  9. Database - pay attention to isolation levels. keep stored procedures to a minimum.
  10. Be ware of operating system and disk I/O limitations such as have a large number of files in one directory.
  11. Synchronous logging

The Importance of Poka-yoke (Mistakeproofing)

Software quality does come from testing alone. Much of quality involves good design practices. One design practice that can’t be forgotten is mistakeproofing. The japenese have a term for this - Poka-yoke.

One of the biggest mistakeproofing challenge is in creating good APIs whether for internal use or external use. We may not even be thinking we’re creating APIs, but any time you create a class or method that another developer has to use it’s an API.

Here are some examples that come to mind:

  1. getTaxRate(String country, String state) - it’s possible to pass these parameters in the wrong order.
  2. Connection dbConnection = new Connection(String longComplicatedConnectionString) - I hate this one, I often have to look at the docs to figure out how to construct the connection string.
  3. callRemoteMethod(String nameOfMethod, Oject[] params, long timeoutperiod) - many problems with this… but I’m thinking of “what if I use a timeout period that’s too short for a particular call?”
  4. Any time I have to call a series of methods in the *right* order.

There are endless examples of APIs that are easy to misuse. Spend some time and you’ll easily be able to come up with a few.

The next time you’re writing software on a team remember that you’re probably creating an API of some kind that has the potential for accidental misuse. Think about how the API could be misused and try to find a way to structure it so that misuse not possible or extremely difficult.

Even Faster Web Sites

To create a great user experience there is almost nothing more important than speed. According to Steve Souders, an expert in this area, 80-90% of the end-user response time is spent on the front-end.

Here is the list of best practices he recommends:

  1. Make fewer HTTP requests
  2. Use a CDN
  3. Add an Expires header
  4. Gzip components
  5. Put stylesheets at the top
  6. Put scripts at the bottom
  7. Avoid CSS expressions
  8. Make JS and CSS external
  9. Reduce DNS lookups
  10. Minify JS
  11. Avoid redirects
  12. Remove duplicate scripts
  13. Configure ETags
  14. Make AJAX cacheable

Watch this Video of Steve’s talk at the recent Google I/O conference:

Even Faster Web Sites (Google I/O Session Videos and Slides)