Object Oriented Database Learning Center

Objectivity Home - Government - Objectivity/DB - Webinars - Download Software

Table of contents:

Object Oriented Database vs Relational Database

Benchmark

We turn now to some benchmarks comparing RDBMS performance to ODBMS performance. The goal here is to search for the relational wall, and, if found, quantify it. This will show when it occurs, how quickly it rises, how large it gets, etc. The benchmark will also show some typical RDBMS operations and investigate how these behave under an ODBMS, in order to determine if anything is lost. In short, the results clearly show the existence of the relational wall, rising exponentially as complexity increases, to orders of magnitude slower than the ODBMS performance. It also shows the ODBMSs are comparable on simple data and simple RDBMS operations, but, again, become far superior, even in joins, due to internal relationship optimization, as complexity increases.

Any report on benchmarks must provide two caveats. First, the user is ultimately interested in his own application performance, not in that of the benchmark. Since ODBMSs address a far wider variety of applications than RDBMSs, this is even a larger caveat here. The best is for the user to benchmark his own application (or find a similar user that has benchmarked a very similar application). Second, benchmarks can become a battle among implementers, rapidly diminishing their relevance to users. Given any benchmark definition, any vendor can go to great lengths to optimize the implementation of that benchmark, even to change the kernel of his DBMS appropriately. Some of this has happened with the well-known TPC benchmarks which, in any case, lack measurements relevant to many ODBMS applications. In short, the benchmark game becomes a competition among benchmarkers, and a competition in which user relevance is lost.

To avoid these problems as much as we are able, we do two things. First, we chose a benchmark content that is generic. It is a model of objects connected to objects, measuring times to create such objects, traverse them, select them, query them, etc. Such a model might be viewed as a very common manufacturing problem, calculating a Bill of Materials on a parts explosion. At the same time, it can be viewed as a very common network management problem, modeling dependencies, fault reports, etc. Other applications include the World-Wide Web of HTML objects and hyperlinks; Inter and Intranets; document management systems, with documents containing possibly shared sections, chapters, paragraphs, and figures; financial services models of users, instruments purchased, and dependencies on other instruments and markets; multimedia systems with video clips and subclips; healthcare systems with patient records, graphs, x-rays, and relationships to doctors, hospitals, procedures; insurance systems with customers, policies, brokers, agents, dependencies, offices; etc. While no claim is made that this is by any means universal, it certainly is relevant to a wide variety of applications, and is relevant at a higher level of complexity than just the simple, flat operations of RDBMSs. Further, it is easy to instrument such a model and show its behavior as the complexity increases, simply by increasing number of parts, fanout, levels of depth, etc.

Second, we have contracted an independent third party to implement the benchmark. Rather than relying on vendors to execute the benchmark, this provides a level playing field in terms of hardware, network, environment, and even level of depth of work in optimizing and customizing the benchmarks. The implementers chose, Objex, Inc., which was lead by Jeff Kennedy. They have over 20 person-years of experience implementing and optimizing implementations using the same, leading RDBMS. Often, they have followed the vendor's own consulting team to further improve the customer's environment. This was chosen specifically so that there would be no question of ample RDBMS experience, so that the results would be fair.

In fact, the implementation for the RDBMS used the highest performance interface of the vendor (via the C language), and most of the well-known optimizations. For example, the database was split across three concurrently accessible disks, one for the data, one for the index, and one for rollback segment. Also, the buffer cache was increased, and the table storage parameters adjusted to accommodate the expected size in the benchmark. Within reason, typical of most users, best efforts were made for the RDBMS implementation. The ODBMS implementation was a straightforward C++ implementation of the model, using a single disk, and no special or unusual optimizations.

The consultants who performed the benchmark report:

"The installation and execution of the benchmark was on a standard Intel Pentium system running Microsoft Windows NT 4.0. The benchmark application was not run in a client/server mode, but instead was run with the benchmark applications running on the same machine as the databases. Parallel versions of the application were developed to access Oracle and Objectivity. Care was taken to make sure that the benchmark implementation did not favor either database, and in particular the timing component was written to eliminate any prejudice that the function might cause to the results.

The results were very surprising. Objectivity outperformed Oracle in virtually all of the tasks at all depth levels except for small numbers of updates and deletes. As the depth increased in all of the tasks, Objectivity exponentially beat Oracle. In fact, the results seemed almost too hard to believe and the tests were rerun and corroborated using a separate program to make sure that we were in fact seeing what we thought we were seeing and that the data was indeed changing correctly. At low depth levels the results were mixed. Oracle was faster at updates and deletes while Objectivity was faster at inserts and connect/selects. At moderate or high depth levels the results showed Objectivity as being up to 30 times faster than Oracle. Basically, as the number of records processed increased Objectivity showed superiority in every task.

As veteran RDBMS application developers, we were not expecting Oracle to be so thoroughly beat. We truly expected the results to be more mixed with Oracle having superiority at such tasks as updates. With very little volume of activity, Objectivity shows its superiority in handling the activities that comprise the parts explosion benchmark."

The hardware configuration used was:

Pentium 150 processor
32MB of memory
3 disk drives (2 SCSI with PCI controller and FAST-ATA 2GB E-IDE)
Windows NT 4.0
Visual C++ 4.0

Both versions of the benchmark program create a set of Categories that has Part trees associated. The benchmark was designed to simulate a bill of materials parts explosion application in which several parts may be combined to form a single part as in the following diagram:

Benchmark Model