Are Benchmarks For Wimps?

But bear with me and do a bit of math:

22 requests per second would be 79,200 requests per hour, or 1,900,800 per 24 hour day.

How many businesses get one or two million hits per day? Not many. For those that do - are they using emulators? No. Are they using crippled samba connections to access the files on some Windows machine through a unix machine? No. Do they usually have about 64mb or 128mb of memory on the 2 million hit servers? No. Do they usually have a 66 or 100 MHz bus speed using an older outdated processor? No.

You might be wondering - what is the point of these questions? Pay attention carefully:

How many businesses get one or two million hits per day? Not many.

The benchmark was done under biased conditions not in our favor, and yet most businesses should find it satisfactory for their website. Most benchmarks are done under biased conditions in the author's favor.

Take into account some more considerations:

• if you aim for 4340 hits per second, and your customer only needs 80,000 hits per day - pull out your calculator. And remember that there are 60 seconds in a minute, and 60 minutes in an hour.. don't forget that extra 60 when you press your little buttons. Some people miss that 60. Some speed freaks don't even own a calculator, because calculators aren't for real men.
• What is the cost of "programmer paid by the hour" to get that 80,000 website tuned up for 435 million hits that it will never receive? Take into consideration the stupidity of someone tuning a website for 435 million hits which will have zero effect - and also take into consideration that some websites will cause more electricity to be used up which have a background process always running in memory..
• A hello world application may show that it can handle 4436 requests per second on one day. But what happens if this hello world application is changed to a real application? Seriously...we've seen hello world benchmarks that show something along the lines of "335 requests per second", and then it quickly drags down to "30 requests per second" once a database and some algorithms are added to the application!
• No one (except us) takes screenshots of the "bad" benchmarks. On the internet are screenshots of good biased benchmarks. And sadly, even a bad benchmark still means about 2 million hits per day - which very few companies receive.
• Interpreters tend to be very fast on a small hello world script - but once you have the interpreter run through several include files, several algorithms, template instantiations, along with database connections - everything drastically changes. A compiled application on the other hand doesn't change much when you add more algorithms, since there is no interpreting of the program or processing include files and such things.
• The database becomes a bottleneck quickly. Invest time into tuning the site using caching tricks like Google does. You may even find ways to avoid hammering a database with large connections by using clever caches. (Joda/Ioda experiments will be done, by the way) In fact companies like Google started out using plain CGI and C++ when they were growing - and they still use C++ today. What Google did was tune their bottlenecks.. not their hello worlds! Their trick was the fact that people were searching for the same stuff over and over again - so why not cache it and save common searches statically? This is intelligent thinking, unlike many speed freaks who spend time tuning the irrelevant parts of the server.

Before we consider obsessing over speed - we must do some calculations and ask our customers (or family members, since many speed freaks build home pages for Grandpa which will obviously receive 3 billion hits) how busy their website could realistically and potentially become.

Most startup companies think they'll receive 1 billion web site hits in only minutes after they launch. In fact it's more like 10,000-500,000 hits per 24 hours, on a good day. What's "good" is extremely relative. For a website like Ebay, Yahoo, or Google: it is obviously very different than 99% of websites.

There are companies out there who would never see 200,000 hits per day, and yet they build servers and pay programmers by the hour to heavily focus on tuning the website so that it can receive 10 billion hits per day (bugs and memory leaks included, all done in plain C or assembly code). Everyone focuses on the amount of hits per second, without rigorously and critically analyzing actual realistic customer requirements first.

Most benchmarks are designed with a hello world application that runs for a few seconds. How about running the benchmark for a few weeks, to take into account memory leaks, especially in FastCGI where if a program is left running for a month there is no memory garbage collection of each leaky process? Indeed, most people do not even know of these issues since a benchmark is run for only a few seconds or mi nutes, not a few weeks or months.

If a customer tells you that they'd never get over 0.5 million hits per day, then would you still build a website that was capable of obtaining 3 trillion hits per day? Would this 3 trillion capable website work for a few days, and then crash in a few weeks? We have to take into account realistic business and customer requirements before obsessing ourselves with speed.

Benchmarks can never be taken too seriously - saying CGI is slow is a vague statement - almost like saying that "100 miles per hour is slow". Compared to what? A rocket ship going to mars? It depends on the requirements. 100 miles per hour is slow, and it is fast too. It's also relative. Speed is relative. Stupidity is also relative.

More benchmarks to come in the future. This is just an example of why we need to pull out our calculator from our desk and use it.