We Really Need TPC Benchmarks for the Cloud

TPC database benchmarks—which database vendors tune specifically for—are a useful objective comparison for buyers of databases. Unfortunately, there is no such comparison in the cloud, and the current cost/comparison approach used by TPC doesn’t fit the cloud.

Here are the problems:

1. TPC doesn’t include costs that are included in the cloud: Public cloud services bundle the costs of everything into their pricing. TPC eliminates things like: electricity, network connectivity, people to run the service, networking equipment (e.g. switches, cables, internet connectivity, etc.), load balancers, modems, Ethernet cards, etc. The public cloud is really a total cost of ownership, while TPC costs are not. So any cost/performance between onsite and cloud solutions compares apples to oranges.

2. TPC assumes that the expenses included above are paid in advance for three full years. Public clouds use a pay-as-you-go model. To compare apples-to-apples here, you would need to do a net present value (NPV) calculation to account for the time-value of money.

3. And this is the BIGGEST issue. TPC tests assume full utilization of the computer. Everyone knows that in the real world you (a) only use at most 80% of the CPU to accommodate usage growth and extreme peaks in performance; (b) between typical peaks and valleys in the remaining 80% your average usage in the real world is typically 10%-20% of the server’s capacity. The cloud, on the other hand, provides an elastic environment where you only pay for what you use. If you use a cloud-ready database that scales elastically, an average load factor of 10%-20% per server translates into saving 80%-90% of the costs versus a dedicated machine. In other words, an elastic cloud environment should reduce cost/transaction by 80%-90%, relative to a dedicated machine.

The cloud provides a normalized cost structure that reflects a more realistic total cost of ownership (TCO). It bundles costs like networking, personnel, electricity, internet access, and more, all in a pay-per-use model. But most importantly, we can get a transactions/instance, and then we elastically scale the instances as needed. This elastic pricing model gives us a real world cost scenario, instead of assuming that a single server is utilized at 100% capacity, which never happens in the real world.

For these reasons, I would like to see TPC create a category of benchmarks that measure cost/performance on standard cloud infrastructures.

Here is a link to Nobu, who ran TPC-C on Amazon’s RDS.