Lack of Business Visibility Cripples Traditional SQL DaaS, Drives NewSQL

More and more public cloud companies are moving to managed cloud services to improve their value-add (price premium) and the stickiness of their solution. However, the shift to a database as a service (DaaS) severely reduces the DBAs visibility into the business, thus limiting the ability to hand tune the database to the requirements of the application and the database. The solution is a cloud database that eliminates the hand-tuning of the database, thereby enabling the DBA to be equally effective even with limited visibility into the business and application needs. It is these unique needs, particularly for SQL databases, that is fueling the NewSQL movement.

DBAs traditionally have insight into the company, enabling them to hand tune the database in a collaborative basis with the development team, such as:

1. Performance Trade-offs/Tuning: The database is partitioned and tuned to address business requirements, maximizing performance of certain critical processes, while slowing less critical processes.

2. System Maintenance Planning: You need to shut down the database to do an application/database upgrade, repartition, etc. but you may need to coordinate with the development team, and the schedule may change up until the last moment.

3. Application Evolution: The database must be designed and tuned to accommodate the planned changes in the application.

4. Consulting with Application Developers: Since the database partitioning and performance are hand-tuned, the DBA must collaborate closely with the application development team on design, development and deployment.

5. Partitioning Requirements: When partitioning (or repartitioning) your database you’ll need to partition the data to suit application requirements avoiding things like cross partition joins, range scans, aggregates, etc., which can cause tremendous performance penalties if not implemented correctly.

6. Moving Processes to the Application Layer: Single server databases can handle joins, range scans, aggregates and the like internally. However, when you partition the database these functions are typically moved to the application layer. The application must add the logic to accomplish these things. It must also add the routing code to point to the correct partitions. As a result, an application written to a single server API does not work in a multi-server configuration.

When moving from a self-managed database—either in the cloud or on premise—to a DaaS, the “DBA-in-the-cloud” doesn’t have that visibility into the business requirements, performance requirements, development schedule, and more. This lack of visibility turns the already challenging task of hand-tuning the database into a near impossibility using traditional databases.

A real world example: You are the DBA-in-the-cloud. Your customer has been running on a single server, but they need to now scale-out across multiple database nodes to accommodate growth. How do you split the data? You don’t know which queries have higher performance priority. You don’t know the development plan for new version of the application. You don’t know when it is convenient to shut down the application to implement the partitioning. You need to inform the application developers how they should implement their routing code to send database requests to the right nodes. You need to inform the application developers that they need to handle joins, range scans and aggregates at the application level, since the database can no longer handle those.

DBAs have enough of a challenge scaling and maintaining databases when they have full visibility into the business. To address these challenges without that level of visibility is unrealistic. I refer to this problem as the “Blind DBA” challenge, because the DaaS DBA has a serious lack of visibility into the requirements and inner workings of the company.

ScaleDB is a NewSQL database uniquely designed to handle the Blind DBA challenge inherent in DaaS implementations. ScaleDB is based on a shared-disk architecture that scales in a manner that is invisible to the application. It eliminates the need to push functions (joins, range scans, aggregates) up to the application layer. It eliminates the need to coordinate your application and database design to work around partitions, because there are no partitions. It eliminates the performance tradeoffs in partitioning design, you get consistent performance across the database. It eliminates scheduling and coordinating application shut-down to repartition, because it doesn’t use partitions. In short, with ScaleDB the DBA needn’t have any visibility into the business to deliver optimal database performance via a single API. This is what makes ScaleDB an ideal solution for cloud companies implementing the database as a managed service or DaaS.

Cloud DaaS Managed Service Fuels NewSQL Market

As public clouds are commoditized, the public cloud vendors are increasingly moving to higher margin and stickier managed services. In the early days of the public cloud, renting compute and storage was unique, exciting, sticky and profitable. It has quickly become a commodity. In order to provide differentiation, maintain margins and create barriers to customer exit, against increasing competition, the cloud is moving toward a collection of managed services.

Public clouds are growing beyond simple compute instances to platform as a service (PaaS). PaaS is then comprised of various modules, including database as a service (DaaS). In the early days you rented a number of compute instances, loaded your database software and you were the DBA managing all aspects of that database. Increasingly, public clouds are moving toward a DaaS model, where the cloud customer writes to a simple database API and the cloud provider is the DBA.

If the database resides in a single server and does not require high-availability, providing that as a managed service is no problem. Of course, if this is the use case, then it is no problem for the customer to manage their own database. In other words, there is little value to a managed service.

The real value-add for the customer, and hence the real price premium, is derived by offering things like auto-scaling across multiple servers, hot backup, high-availability, etc. If the public cloud provider can offer a SQL-based DaaS, where the customer writes to a simple API and everything else is handled for them, that is a tremendous value and customers will pay a premium for it.

While this sounds simple, public cloud companies soon learn that the Devil is in the details. Managing someone else’s database, without insight into their business processes, performance demands, scaling demands, evolving application requirements, and more, is extremely challenging and demands a new class of DBMS. These demands have created a market need that is now being filled by companies using the moniker “NewSQL”.

In short, when it comes to DaaS, public cloud vendors want the following:

• Simple “write to our API and we’ll handle the messy stuff like scaling, HA, etc.”

• Premium value that translates to a higher profit margin business

• Barriers to customer exit

Future posts will delve into the operational demands of DaaS, and how these demands a driving NewSQL DBMS architectures and features.

HP Needs a Linux OLTP Database...FAST

Oracle, after dating HP, Dell, Netapp and EMC has found its mate in Sun. Oracle is now becoming a systems company, and unceremoniously dumping these former paramours. These leaves the spurned lovers to find alternate accommodations, especially in the area of the database.

As I have stated previously on this blog, the clear partner of choice on the Windows front is Microsoft. This is demonstrated by today’s partner announcement around MS SQL Server for OLTP. But who is their partner in the Linux segment?

The following are contenders:
* Postgres (HP rolls their own)
* EnterpriseDB (pre-rolled Postgres)
* Ingres or Sybase—Oracle has felled them both in the past, but they are hoping for new life with a big sugar daddy like HP.
* ScaleDB, If HP is going after the cloud and the MySQL market

I don’t see them going for a NoSQL solution because NoSQL = NoEnterprise, making it a non-starter for HP. One way or the other, HP needs a solution for OLTP on Linux and they are on the clock.

For OLAP, HP has NeoView. If they felt the need, there are a number of OLAP solutions out there such a Greenplum, Netizza, Asterdata, Paraccel, Ingres/Vectorwise and others. That said, I think HP feels that they are holding a good hand on in the OLAP space, but Linux-based OLTP just became a gaping hole in their product suite. Today's partnership with Microsoft confirms this problem, but only solves the Windows half not the Linux half.

Oracle/Sun vs. The Cloud

Larry Ellison makes it very clear that Oracle believes in a back to the future model where software and hardware meld together into “systems”, purpose-built, integrated solutions. In other words you won’t buy an Oracle database and a server and configure it to run a data warehouse, instead you’ll buy the “Oracle Data Warehouse Server.” The first such system is Exadata, which is apparently doing quite well, according to Ellison.

This is a classic bundling, although some may call it a tying strategy. Microsoft, seeing that they couldn’t win each office productivity segment individually—including word processing, spreadsheet and presentations—decided to play to their strength and bundle them into a solution that no individual company could compete with. This is bundling. The tying strategy is where Microsoft used their dominance in the operating system to tie the browser to the OS, thereby owning the browser market. In the case of Oracle, one could make a case either bundling or tying. I’m making neither a value, nor a legal judgment about Oracle’s strategy; I am just providing historical context.

Ellison points to Cisco and IBM, under T.J. Watson Jr., as examples of successful systems companies. But my question is simple: Will this back to the future strategy work against the cloud? Assembling solutions with pre-packaged systems is certainly easier than starting with more granular components like hardware and software. But does it really stack up against today’s benchmark, the cloud.

Let me use a transportation analogy:

Assembling all of the components (hardware, software, etc.): Like building a car piece by piece

Assembling systems (a la Oracle's Exadata and Cisco): Like building a car by installing large grain items, the chassis, wheels, engine, etc.

Using the cloud: Like buying a pre-built car off the lot

SaaS Applications: Like riding the subway

Most people are perfectly happy either buying a car or riding the subway. For really high-end performance, some may want to build their own car with components or by hand, but it’s a relatively small market.

I don’t expect any public cloud offerings to satisfy high-end enterprise demands…yet. But I have to admit, the cloud is evolving quite rapidly. Just look at Amazon and their introduction of Virtual Private Clouds, Elastic Block Services (a SAN in the sky), Boot from EBS, etc. I can launch an entire cluster with a mouse-click, without talking to IT. How can you beat that? Historical precedence is also on the side of commodity technologies, like the cloud, growing up to cannibalize the high-end. The PC cannibalized the workstation, which cannibalized the mini, which cannibalized the mainframe. From the clou's perspective, the trend is their friend.

The cloud won’t seriously threaten large enterprise systems for quite some time, but I believe it is just a matter of time. Oracle can certainly ride a strong wave of current demand for systems. I expect that in time they will also provide a compelling suite of solutions in the cloud. But if I were a bettin’ man I’d have to bet on the cloud; they have simplicity and history on their side. On the other hand, it is hard to bet against Ellison.

Virtual Databases: The Face of the New Cloud Database

Shared-disk databases can be virtualized—making them cloud-friendly—while shared-nothing databases are tied to a specific computer and a specific data set or data partition.

The underlying principle of the shared-nothing RDBMS is that a single master server owns its specific set of data. That data is not shared, hence the name shared-nothing. Because there is no ability to share the data, there is also no ability to virtualize the computing of that data. Instead the shared-nothing RDBMS ties the data and the computing to a specific computer. This association with a physical machine is then reinforced at the application level. Applications leveraging a shared-nothing database, that is partitioned across more than one server, use routing code. Routing code simply directs the various database requests to the servers that own the data being requested. In other words, the application must know which server owns which piece of data. This further reinforces the mismatch between shared-nothing databases and virtualization.

This is not to say that it is impossible to virtualize a shared-nothing database. As any software architect will tell you, “You can do anything in software…” The second part of that statement is “…but it may not perform or scale well, and it may make maintenance very painful.” The latter part of that statement is exactly what you will find with any effort to virtualize a shared-nothing database. Attempts to insert layers of indirection will result in added complexity that makes maintenance a nightmare. Finding bugs, tuning performance, recovering from failure, all of these issues are severely compounded when you introduce layers of indirection in a shared-nothing database.

The performance, and hence the scalability are also undermined in this model. In order to support dynamic virtualization, you must mediate the requests from the application before they hit the database. This requires a piece of middleware that sniffs each database request and routes it to the appropriate server. What happens when a database request spans multiple servers? Suffice it to say it isn’t pretty, and it doesn’t perform well. This sort of request will result in a lot of data shipping and joins. The bottom line is that partitioning your database to achieve performance, scalability and maintenance is a black art, all attempts to automate this process have failed.

Compare this to the shared-disk DBMS. Shared-disk separates the compute from the storage. The data is stored in one big trough, while you can have any number of compute instances feeding on the entirety of that data. Because each node has access to all of the data, you don't need any middleware to route the database requests to specific servers. Furthermore, each of the compute nodes is identical, making them virtualization-friendly. If one node fails, the others recover the transactions, while the application continues uninterrupted. You can also add nodes on the fly, again without interrupting the application. For these reasons, the shared-disk RDBMS is ideal for virtualization, while the shared-nothing RDBMS is anathema to virtualization.

This is an excerpt from a white paper I'm writing that addresses virtualized cloud databases.