Blog by Objectivity

About the EclipseCon developers' conference

noreply@blogger.com (Objectivity) — Fri, 02 Apr 2010 17:58:00 +0000

Objectivity was a sponsor of this year's EclipseCon developers' conference in Santa Clara, California. The event ended last Thursday, and I'm back at headquarters sharing our experience with the rest of the company (and now, you).

EclipseCon is primarily focused on users of the Eclipse Modeling Framework (EMF) and some of the interesting projects related to EMF and the Eclipse Foundation. While the Eclipse Framework is the result of an open source community effort, the tools themselves are used throughout the entire spectrum of developers, from basement "tinkerers" to enterprise technologists responsible for building some of the most advanced systems on earth - and elsewhere.

From the enterprise to outer space.

NASA is using the Eclipse Framework and Connected Data Objects (CDO) to build numerous systems related to their exploration of Mars, including vehicle control systems, mission planning and other operations. Leading enterprises including Echostart are also using the EMF/CDO in their mission-critical systems as well.

Representatives from NASA, Echostar and many other organizations came to see us at the conference. They had heard about the upcoming availability of our CDO-pluggable data store. We've announced this in recent press releases, but we were also mentioned specifically by leaders of the Eclipse Foundation during several of their sessions with developers at EclipseCon.

Why is the Objectivity/DB data store for Eclipse CDO generating such interest within the Eclipse community, and with enterprise developers using Eclipse and CDO? Three words: Performance. Performance. Performance.

Until now, developers using the CDO Model Repository could use a relational database and another layer like Hibernate to map Java objects to relational tables. This O/R mapping and translation can have significant impacts on performance. Objectivity/DB natively supports objects and object-oriented languages like Java, and eliminates any need for a mapping layer. Not having to translate objects reduces the application complexity and greatly improves performance.

Add to this, the facts that (1.) Objectivity/DB in uniquely distributed -- enabling a single logical view or access between the application and data sources distributed across tens of thousands of nodes, (2.) that we can cache or persist complex relationships and queries, allowing instant access to the results, and (3.) that object technology like ours naturally supports real-time traversal of graphs (see "graph theory" on WikiPedia), and you can see why enterprise developers, technologists and decision makers were interested in getting more information from us.

Opportunities to leverage innovation from the open source movement and commercial products.

The funny thing about EclipseCon and similar events that promote various open source tools and components, is that some people (on the outside) assume there's no place for commercial products in this environment. As we spoke to developers and leaders of the Eclipse Foundation themselves, everyone confirmed for us again, that the future of application development of mission critical systems is dependent on both open source and commercial elements. Top public and private organizations, business interests, solution providers and enterprises are all looking for solutions that involve several moving parts -- and they are absolutely looking to leverage the best components available from any source. These solutions include innovations not only from the open source movement, but also any commercial components that are best suited to their requirements.

Take aways from EclipseCon 2010

The Eclipse Foundation and CDO Model Repository Project are providing Java developers with a comprehensive IDE and set of tools and components they can use to build mission-critical and enterprise-level solutions, and to more easily select from and utilize a much wider range of components - including a commercial solution like ours. Today, one size does not fit all. If your requirements and objectives are best achieved with a particular solution, then your organization would be best served by using that solution - whether it is open source or commercial.

We had a great time, and welcomed the opportunity to connect with leaders of these open source projects, enterprise decision makers and technologists. The leaders of the Eclipse and CDO projects, including Eike Stepper and Ed Merks, are true professionals doing great things in support of the Eclipse ecosystem and developers looking to leverage the best from all possible sources. By the way, we recorded some video interviews with them at the event, and will have them posted to the site shortly.

Thomas

Thomas Krafft is the Director of Marketing and Demand Generation at Objectivity.

Thank you Google! (for talking about graph data and computing)

noreply@blogger.com (Warren Davidson) — Mon, 15 Mar 2010 22:31:00 +0000

We all know that Google hires lots of really smart people. Why? Because they can of course. They treat them well too, I dined at their cafeteria and I can tell you there really is no reason in the world a Google employee would actually leave the campus for lunch.

When Google's smart people told the world what they did with MapReduce as a way to manage big data, lots of people started looking at this and realized they could use MapReduce too (although you do have to be a smart cookie to do what they do, but that's another story). Now Google is talking about Pregel and graph data, which we find even more interesting.

What is Pregel? Pregel is used for processing graph data, where data is presented as 'nodes' and 'edges'. Think of the "6 Degrees of Kevin Bacon" exercise, where everyone would be considered a "node." If one person were to send an email to another, that would be an example of an "edge" between two nodes. If I know two other people, and the one I emailed also happens to know Kevin Bacon, there's data showing a 2-degree separation between myself and Kevin Bacon. A graph could easily find the links between large numbers of people by simply traversing these nodes and edges. But graphs can be used for much more than just finding your connection to a movie star. Applications include finding bad guys, fraud detection, analyzing gene sequences, geo applications, traffic distribution and much more. We even see the area of BI (business intelligence) and Knowledge Management moving towards this need, as large organizations start to see the real value of relationships.

Google says that MapReduce covers 80% of their data processing needs, and the other 20% is handled by Pregel, which was designed to determine relationships from graph data. Some of our customers are the same kind of smart people at Google, you know -- the scientific type. They have been building applications to traverse vast graph data on Objectivity/DB for many years now. We know there is a need for this solution in the market, and we're glad that Google is helping contribute to the discussion on possible solutions. It's good for the market, and it's good for us. And just like other solutions, one size does not fit all. Pregel will address some needs in the market, and Objectivity/DB addresses others.

The problems that could be solved with a graph data solution are becoming almost too numerous to count. It's becoming a common issue that most developers don't know is much easier to solve today. As more developers discover that "walking the graph" is actually becoming a relatively simple process, their use of graph data almost instantly becomes ubiquitous within their organizations and applications. There are so many ways in which real-time traversal of graph data could benefit their organizations.

Anytime you need to find a path or connection between objects, graph solutions are the way to go. And we've seen how important the data store is to this problem. In a relational system you have to build a complex series of self-joins, which just kills an RDBMS's performance. Relational technology was never designed to follow relationships. Relational databases are designed to catalog and retrieve tabular, structured data. If you want to sort a billion numbers quickly, relational is the way to go. But if you try finding relationships between objects separated by more than a single degree, the process quickly and exponentially begins to bog down and degrade. In an internal test, Objectivity/DB traversed a relatively small number of relationships (96,000) to 5 degrees, in 15 seconds. It's also important to note that we began returning results to the user immediately and filled in new results as they became available -- rather than making users wait until the entire process has completed. We ran the same test against a leading commercial relational technology, and had to wait more than 16 hours for the process to finish.

This is why Google is building Pregel to address a particular set of issues common to their domain, and it's the same reason many of our customers have used Objectivity/DB to solve graph problems in their domains.

It's always great to have a company like Google helping educating the market on various topics, such as graph computing. They have a much bigger soapbox from which to promote different technologies, and in the end, the more public awareness, the better. People will find and use the solution that works best for them, and we're excited to be answering all of the graph data questions we've been getting in recent months.

So, thank you Google!

Eclipse (EMF), Cloud Computing and Big Data

noreply@blogger.com (Warren Davidson) — Thu, 11 Mar 2010 15:05:00 +0000

We've been doing a lot of prep recently for our upcoming participation in the EclipseCon 2010 in Santa Clara California this month. We're excited to be opening the door for many more people to see how Objectivity can be used in their application and systems development.

We've been working on an Objectivity Eclipse CDO (a plugin for the Eclipse Modeling Framework or EMF), and the nice thing about the CDO is that people will be able to work with our software in the comfort of their own environment.

And then we have the upcoming Cloud Expo East 2010, which should be a quite a blast out in New York. We're working very closely with numerous distributed cloud computing platforms and service providers, providing them the first commercial federated, distributed data management and object persistence solution available anywhere. And of course we've been having a lot of conversations around NoSQL, graph data and big data.

It led me to wonder about the synergy between the two communities. Our CDO plug-in we know will be great for people with complex data models. And we've often been the company used for big data (highly scalable, distributed data supporting real-time application requests. Then I ran across a blog by Ian Skerrett who is Director of Marketing for the Eclipse Foundation and has a very nice Eclipse community blog. A couple of weeks ago he blogged about the need for big data and Eclipse. If he is correct, and I believe he is, people with big data requirements are looking for better tools and solutions like ours.

We're excited to be working with the Eclipse Foundation, and hope to learn more about how the Eclipse community is working with big data. If you plan to attend either of the industry events mentioned above, stop by and let us know what you think. These shows are great ways to get feedback directly from the user communities, and we love the input.

Haiti Earthquake, Emergency Preparedness, Disaster Relief and Technology

noreply@blogger.com (Leon Guzenda) — Fri, 15 Jan 2010 18:29:00 +0000

When the earthquake struck in Haiti the phones and electricity went out almost immediately. We're all used to being connected around the clock with our cellphones, text messaging, email and web access, but most people on Haiti don't have those tools at the best of times. However, some cell towers stayed up and within minutes people were posting messages to Twitter and Facebook and using Skype to contact the outside world.

Most of Objectivity, Inc.'s employees live and work the California earthquake zone, so we have a disaster preparedness plan that we've refined over the years. This includes an alternate, emergency web site hosted outside of California, where our employees and customers can communicate and receive instant status updates and uninterrupted support for any systems effected by the emergency.

Objectivity, Inc.'s headquarters weren't badly affected by the Loma Prieta earthquake in 1989. We shut the office for a day and waited for the city to declare the building safe, but essential services were restored within hours. It was a 7.0 on the original Richter scale, or 6.9 on the moment magnitude scale, which seismologists prefer now. Our partners in Japan weren't as lucky when the Great Hanshin earthquake struck Kobe in 1975. It was a 7.2 and it caused widespread destruction there. We were able to communicate with the Tokyo office, but it was almost a day before the safety of their colleagues in Kobe was confirmed.

Coincidentally, an Objectivity customer, "Bolt, Beranek and Newman" (now BBN Technologies), had built an advanced logistics and remote collaboration prototype for the DoD and it was undergoing trials at several Command and Control Centers and the Red Cross. Teams were able to share plans, action lists, images, maps and so on and the system played a useful role in co-ordinating the efforts of multiple organizations trying to get assistance into Kobe.

Objectivity/DB was also used in an operational logistics system for the Tokyo transportation network, allowing them to plan around construction work and emergency situations. The ability to handle complex networks, geospatial information and other complex data makes Objectivity/DB ideal for logistics application.

I mentioned the role of SMS, Twitter and so on at the start of this article. I was pleased to see that the Red Cross can now accept donations via text message, as well as their main site. They've set up a system that allows people to text HAITI to 90999 to contribute $10. That's a great idea, as every little bit helps. As of this morning (01/15/10) they're reporting that they've received over $8 million via that new channel. Isn't technology great?

MySQL Creator Wants To Save It From Oracle

noreply@blogger.com (Leon Guzenda) — Wed, 30 Dec 2009 17:00:00 +0000

Michael "Monty" Widenius recently launched a campaign to save MySQL from being driven into the ground by Oracle, which is in the process of buying the open source RDBMS product's previous owner, Sun Microsystems. Widenius is hoping to rally users to canvas the European Union opposing the Oracle deal. He argues that the code base can survive with renewed effort on the part of the community. He founded one such initiative, called MariaDB.

He's probably right that it could survive, but one of the key reasons that MySQL did so well is that, like Red Hat, a commercial venture stepped in to give large enterprises the support that they thought they might need. Opinions were all over the map when Oracle purchased it and the list of pledges that they've given the EU don't actually amount to much when you consider the obligations that accompany any licensed product. However, it's very likely that the EU will approve Oracle's bid soon.

They won't have the playing field to themselves though. Ingres is mounting an aggressive attack on the MySQL base and PostgreSQL has an extra lease of life, especially after EnterpriseDB and some data warehousing ventures adopted it.

There's also a growing recognition in the IT community that RDBMSs are no longer the de facto answer to every one of their data management problems. The rise of the "NoSQL" movement is testimony to that. People forget that, even after more than thirty years, relational database hold less than 20% of enterprise data and significantly less of the total amount of data in the world. Most of it is in structured files.

MySQL helps a particular group of developers solve some particular types of problems. For that reason alone, every technologist would like to see MySQL survive what will likely be a difficult journey in coming years. At the same time, I have to point out there are many other needs that relational databases cannot satisfy.

Objectivity/DB can handle all types of data, whether structured or unstructured, much more easily than relational databases. Objectivity/DB is designed to support more than a yottabyte of data addressing space (note that this is more than all the data in the world today), while both Oracle and MySQL still struggle to effectively manage tera and petabyte volumes. And it helps developers solve problems that would be very difficult, if not impossible, on other DBMSs - such as discovering many-to-many relationships in near real-time (link analysis), and distributing data without significant engineering and IT overhead (which is ideal in cloud computing environments, for example).

And if that weren't interesting enough, "Release 10" will be available soon after the holidays. Watch this space for more details, coming soon...

Supercomputing 2009 - Higher Performance Computing

noreply@blogger.com (Leon Guzenda) — Fri, 04 Dec 2009 17:59:00 +0000

Warren Davidson and I attended Supercomputing 2009 (SC'09) in Portland, OR in mid-November. We didn't see any breakthrough technologies this year. Exhibitors were mostly showcasing incremental changes to their products.

Overall Impressions

The large DBMS vendors were noticeable absent and some of the largest storage vendors were missing too. That may be a sign that the recession is causing people to use their marketing budgets more carefully.

However, Supercomputing isn't really a sales venue for us. Most of the potential customers there are already Objectivity/DB users, so it's more of a venue for networking with customers and partners, plus an important opportunity to see what's going on in the HPC community.

Some companies, such as SiCortex and 3Com had either gone out of business or been taken over since last year's conference. Oracle and EMC were notably absent. Microsoft had a very large stand, as usual, but nothing really new to show in their HPC operating system lineup. There were far fewer independent cluster assemblers than last year.

Display Technology

There are generally a lot of extremely large displays, but they were noticeably absent from this year's show, perhaps because of controlled spending on the part of the exhibitors. The only 3D display, a goggles-free version, was there last year too. We did spot one display that takes up to 8 feeds from separate computers, which is handy in some environments, such as surveillance, simulation and C4IStar. FusionIO/ioDrive had an array of HD screens displaying thousands of discrete video streams being refreshed from solid state storage. It was impressive.

Networking

ICANN had a stand there for the first time ever. They're trying to get the word out that IPv6 is needed and real. They believe that there will be a significant government systems mandate about a month from now. Objectivity/DB Release 10 (scheduled for general availability later this quarter) supports IPv6.

Network bandwidth has been a major theme at all HPC events over the past two decades. The SciNet consortium had a very impressive array of equipment (worth $20M) demonstrating 40 and 100 Gigabit Ethernet and 12x QDR Infiniband. The WAN Bandwidth was 400 Gigabits per second - that's 200,000 times more than an average home connected to broadband and would be capable of transmitting 100 Blu-Ray movies in one minute. Several stands had audio/video links to remote locations, notably Cisco and the University of Georgia, the latter offering an opportunity to talk with a diver in an aquarium in Atlanta.

Processors

NVidea was the only major GPU manufacturer there last year. This year there was at least a dozen companies showing hybrid systems that combine conventional CPUs and GPUs or software to support highly parallel, hybrid environments. GPUs tend to help a lot with streaming data or highly repetitive, simple calculations, such as conversion from one architecture to another. IBM, HP, Dell, Cray and Penguin Computing had large stands. ORNL had chunks of various supercomputers in an armored glass case, raging from a feeble 50 Gigaflop board from 10 years ago to a 160 Teraflop board. The current quest is to build Petaflop machines, but a few places are looking at two or three orders of magnitude above that within 10 years.

Software

We saw a lot more parallel compiler development tools than last year. The price is dropping to around $600 per seat too. I also noted some interesting open source alternatives to Hadoop, such as Sector. Objectivity/DB is a distributed DBMS, so there's not much opportunity to parallelize much more in the kernel and all of the servers can exploit multi-core CPUs. However, parallelization tools may be of interest to application developers, so I'll watch out for good ones. If you're already using one, please let us know via the Comment link below.

Grid and Cloud Computing

There are a lot of cloud and grid dashboard companies out there and about half of the grid specialists now have a cloud story. Most of the major players, apart from Cloudera and GoogleCloud, were there. Objectivity/DB is now available on the Amazon EC2 Cloud, of course.

Storage

DDN had a huge stand, but, as I said above, other than solid state storage, IBM and HP, the storage industry was largely MIA. As has been the trend for the past decade, most advances were in storage density rather than random access or steaming speeds. It's odd to realize that a Terabyte of IBM RAMAC storage would have cost about $100 billion in 1957, versus around $100 now, but the access speeds are still about the same.

MapReduce implementations are not a low-latency solution

noreply@blogger.com (Todd Stavish) — Fri, 20 Nov 2009 05:35:00 +0000

It's important to note that MapReduce implementations are not a low-latency solution. In fact, MapReduce is not particularly adept at being "query-able." There is generally something on-top that makes it so, Google has BigTable on top of their map-reduce solution, Hbase (Data Warehouse) \ Hive (Columnar) live above Hadoop files (HDFS).

Hadoop is a batch processing system. Accordingly, this system does not and cannot promise low latencies on queries. The paradigm here is strictly of submitting jobs and being notified when the jobs are completed as opposed to real time queries. As a result it should not be compared with systems like Objectivity/DB.

Hadoop thinks in hours. Objectivity/DB thinks in seconds or less.

From an Objectivity/DB application development perspective, Hadoop potentially takes a lot of the grunt work out of processing raw data for us. Namely, parsing unstructured or semi-structured data. SSAFE (a customer implementation for space situational awareness) had to build their own processing pipeline for raw data (Rogue Wave Hydra and custom code). MR may have been moer appropriate for the raw data processing.

If you still want to "walk a graph" ad-hoc in a low-latency, real-time fashion - which is what we do very well - then you'll still want Objectivity/DB.

For Hadoop integration, Cloudera is our choice (Cloudera is to Hadoop, what Redhat is to Linux). The conventional integration points don't work for us, as they are primarily relationally oriented. So no Sqoop (only one-way anyway), nor DBInputFormat (bidirectional but JDBC based).

For us, I think integrating directly via the map and reduce functions is the best approach. Thrift is also a possibility, however as it stands now you have to go through a proxy service, which makes me nervous.

From a scalability perspective, Objectivity/DB could scale side-by-side with Hadoop nodes (i.e. horizontal scaling via commodity hardware). Traditional relational systems are sensitive to the load that Hadoop could place on them. In fact, bulk exports are often used (which means loading delays and potentially stale data) to protect relational processing performance on the live system. Objectivity/DB could stream in processed Hadoop information in parallel as quickly as it was computed and still support low-latency query. That's not too shabby.

Todd Stavish is an Objectivity, Inc. Systems Engineer with expertise in distributed computing applications. He advises customers about complex modeling, performance optimization and building fault tolerant applications.

Cloud Computing Expo - 2009

noreply@blogger.com (Leon Guzenda) — Mon, 16 Nov 2009 22:00:00 +0000

I wasn't able to attend the 4th International Cloud Computing Conference & Expo last week at the Santa Clara Convention Center in Santa Clara, CA, so I've asked Warren Davidson, Objectivity's Director of Business Development, to give us all an update on his impressions of the venue. Here's Warren...

~-~-~-~-~-~-~

Objectivity attended this year's Cloud Computing Expo West with great anticipation. There has been a lot of discussion about cloud computing, and we were particularly interested in learning more about the motivations of the people attending. What were the most important questions they wanted answered? We also wanted to see what other technology vendors were doing, and how people were starting to solve problems in the cloud computing space.

First I should point out our interests. Cloud computing means many things to many people, but one thing we know (and validated at the show) was that many different types of organizations want to take the best from cloud computing and apply it to their products, infrastructure and corporate strategy. For many attendees this meant learning how cloud computing could provide the insight on how to build highly scalable systems more cost effectively. Many want to reduce their complexity, and solve some of the challenges of traditional IT architectures.

And many attendees had mixed interests. Some wanted to learn how they could leverage SaaS offerings to move some of their infrastructure into the cloud, while building in-house systems where needed. There were many vendors there who could help companies with both, such as 3Terra. Others were providing customers the platform to move their infrastructure out to the cloud, like GoGrid and Rackspace. And then some provided the platform to get to these platforms such as Rightscale.

Our focus of course was how to manage data in a cloud, whether it is public or private. We wondered if people saw current database strategies well suited to their needs. In some cases where people had very simple data the answer was yes, something like Hadoop or SimpleDB would do the trick. But for many of the attendees that was only to start, or "Cloud v 1.0" as I call it. Their expectation for the future was they would need something more robust. And many said that their data was complex - using file systems or trying to manage data across relational databases was not promising.

In the end we found that people wanted to start with distributed systems, and reducing complexity is a key issue. This is tied into scalability and performance, but in the end using a fully distributed database system would solve some major problems for many customers. If nothing else many people know they can't truly scale with relational databases, relying on partitioning and sharding to get around short-term data issues.

The show was a good start. Keep an eye on this space. There is a lot more to learn and we'll make sure to post more information on the topic of managing data with respect to your cloud computing strategy.

Warren Davidson,
Director of Business Development
Objectivity, Inc.

It's Easy Going Green With Objectivity/DB

noreply@blogger.com (Leon Guzenda) — Thu, 12 Nov 2009 18:46:00 +0000

IT departments are very aware of the need to save energy, colloquially labeled "going green". We can all do our bit to help by taking a few simple steps, such as changing the energy saving settings on our computers at work and home. I've recently started moving my personal web sites to a "green" web host provider. They guarantee that 100% of the energy they use in their data center comes from renewable resources. Objectivity's headquarters had a smart energy control system to reduce our overall energy consumption. Whether or not you believe in "saving the planet", it makes sense to save money.

If you're an Objectivity/DB user you're implicitly saving energy. When we benchmarked Objectivity/DB against RDBMSs a couple of years ago we needed one quarter of the processor power and 60% less storage than our competitors. Not all applications will show such a large saving, but there is always a CPU saving due to the lack of a mapping layer and a disk storage saving because of the lack of join tables and unnecessary indices. That can translate to better performance and greater throughput. It can also mean running fewer processors to achieve a task.

Objectivity/DB associations (called "relationships" in Java and C#) can take between 8 and 24 bytes per bi-directional link. In-line links can be 4 bytes to other objects within a container and 8 bytes for connections that span containers or databases. Dynamic links are always 12 bytes. Associations are extremely efficient relative to stored join tables and their accompanying B-Tree indices. Scalable collections also use less disk space than their relational equivalents. The more complex the application and data, the greater the saving.

You'll also be saving energy if it takes you less time to accomplish a development or administration task. Just remember to go home early and turn off the lights. Good luck on the going home early part! :-)

Database Transaction Management Paradigms

noreply@blogger.com (Leon Guzenda) — Sun, 08 Nov 2009 06:02:00 +0000

Almost all DBMSs (database management systems) provide some form of transaction management. The most common mechanism enforces a set of properties commonly known as ACID (Atomicity, Consistency, Isolation and Durability). A transaction can be regarded as a single logical operation that either succeeds or fails. If the transaction succeeds it may have altered the state of the database. If the transaction fails it should not have changed anything in the database. If the transaction succeeds and has changed the database then the alterations are persistent. They can only be removed by undoing the transaction, generally by using some kind of database operation (write, delete, update etc.) or "before look" log.

I've posted a series of pages detailing several issues related to transaction management, along with related decisions made in the development of Objectivity/DB, our distributed data management solution - a market leader used in some of the most complex systems in use today. Here are the jump links:

DBMS Design Decisions

noreply@blogger.com (Leon Guzenda) — Sun, 08 Nov 2009 06:02:00 +0000

As with any complex software, designing and implementing a database management solution involves setting goals and making decisions and tradeoffs to meet those goals as well as you can. When we designed Objectivity/DB we set down ten major goals. The first four are relevant to this series of posts, namely - Reliability, Performance, Scalability and Distributability.

Transaction management, particularly controlling the in-memory state of data and the persistent data on disk, is fundamental to achieving reliability. However, writing every operation or database change to a separate log file can degrade performance. That becomes increasingly important with high throughput systems in constrained environments. Imagine a one Terabyte database in a satellite. If I start a transaction that scans through every object, updating a single field, then creating new objects and linking them to the original objects I may have to create a journal that is a Terabyte or more in size. What if I want to do even more in the transaction, such as using the old and new objects to generate even more objects using an iterative algorithm that may cycle through the objects many times? I may not have enough disk space in my constrained hardware environment to complete the transaction.

So, we decided to use a form of distributed, hybrid, shadow paging instead of a conventional log. The technique has actually been used in quite a few DBMSs as you can always predict the maximum amount of scratch space that a transaction will require. Updated database pages are written to a different physical location than the original logical page. The transaction commit operation safeguards the database by writing the old page map and the new page map to a separate log file. If all of the data remaining in memory is successfully flushed to disk and made persistent by calling fsync, or something similar, the new page map can also be made persistent and the log file can then be deleted. If the transaction is interrupted at any point, right up to the final removal of the log file, it can be rolled back by the owner process or another one that has sufficient permissions. The old physical pages can then be returned to a free file space pool. In the scenario I described above, we can write and rewrite the Terabyte of data many times and we'll never need more than two Terabytes plus a small log file.

Dealing with distributed databases slightly complicates the mechanism, as you may successfully update several database files on various machines and then fail to reach another one, so the whole transaction must be rolled back. We decided to use lock servers to help control transactions. If a client process dies while it is holding locks you can set a lock server to automatically rollback the transactions that the failed process owned.

Lock Granularity

noreply@blogger.com (Leon Guzenda) — Sun, 08 Nov 2009 06:02:00 +0000

Performance is always an issue and perhaps the most visible trade-off was the decision to lock bunches of objects rather than individual objects. In an Objectivity/DB federation that translates to locking at the container level rather than the object level. Doing that dramatically reduces the number of lock requests. However, there's a potential downside - a long running update transaction could prevent other clients from reading the objects in the containers it is updating.

The solution was quite simple and it exploits the underlying shadow page recovery mechanism. When you start updating a transaction the Objectivity/DB Storage Manager, a major component of the kernel, creates a new version of the container. Updates and new objects are written to new physical pages. The old ones are untouched until the commit occurs. The introduction of the Multi-reader, One-Writer (MROW) transaction mode makes it possible for existing or new readers to access the previous version of the container while the new one is being written. In some cases the new version will be rolled back, freeing the old versions for updates by subsequent transactions from any client.

The picture is made slightly more complex because of the need to maintain a consistent view for long running readers. Suppose that process A opens version 1 of a container. Then process B starts updating it, creating version 2. Then process B commits the transaction while A is still running. Process C then opens the container and it sees version 2. That's not a problem unless processes A and C have to collaborate in any way that involves this container, as they are looking at different versions. When process A eventually finishes its transaction there will be two versions of the container, but only the latest one will be picked up by new readers. This mechanism can cause some version creep, with multiple versions of a container existing in the database file. To prevent that being a problem, the kernel deletes the old, not currently accessed, versions as soon as a new updater comes along.

Object server DBMSs generally log individual operations or page updates and lock at the object level. The problem with that approach is that you not only have more lock server traffic, you also run a big risk of deadlocks as clients navigate through complex networks of objects. The subsequent queueing can significantly degrade performance. I was very amused at the recent ICOODB conference to hear an engineer from another ODBMS supplier describing that object level locking within their system data had been causing bottlenecks and limiting server throughput. He then proudly described the system of latches (transient, within transaction locks) and data redesign that had allowed them to defer, but not eliminate the problem. What he was saying, out of the earshot of their sales and marketing people, was that their highly touted object level locking was actually hurting the performance of their server.

Every time that we assembled a list of desired features for new releases in our early days, the sales people would request "Object Level Locking." However, our users were finding that the performance of our distributed MROW implementation was much better than centralized servers with object level locking.

Locking Semantics

noreply@blogger.com (Leon Guzenda) — Sun, 08 Nov 2009 06:02:00 +0000

Objectivity/DB has a hierarchical storage model, with the federation owning databases, which own containers, which own objects. Or, looking at it from the bottom up - objects are in containers in databases in a federation. When we looked at the locking paradigm it was immediately obvious that the best fit would by an implementation of Gray Code hierarchical locking. It adds Intention lock modes to the conventional Read (called Share) and Update (Write) locks.

Locking something automatically results in locking both higher and lower level items in an appropriate mode. When you open a federation in "Read" mode you are not blocking other users who want to update databases and containers in that federation. You actually have an IS lock on the federation. The lock monitor utility will show you that lock. If you have an OID and you open that object in "Update" mode the kernel will automatically obtain an IX lock on the appropriate database and an X lock on the container. The IX lock warns other processes that somebody is updating something in this database, but it doesn't prevent them from opening other containers in either read or update mode. The rules become slightly more complex when MROW transactions are involved and you may see C (for Concurrent) and IC locks in addition to X, S, IX, and IS.

If you want to make absolutely sure that you have safely updated a group of related objects in multiple containers you can use lock propagation to atomically obtain the update locks on all of the containers. The locks are propagated along associations, which may be between an owning container object, such as the one for a Document, and the underlying Chapter containers.

Hierarchical locking, particularly in a distributed environment, isn't easy to implement, particularly when you bring deadlock detection into the picture. When I first met Dr. Jim Gray he was very pleasantly surprised to hear that a commercial DBMS was successfully applying his ideas. One very nice side effect of this mechanism is that it dramatically reduces the amount of locking code that you see in an Objectivity/DB application. It happens implicitly and safely when you touch or update an object. When we convert applications from other ODBMSs we have to remove all of the extraneous locking code and simply make sure that the methods that set the contents of an object's fields all apply the update() method.

Novel Transaction Modes

noreply@blogger.com (Leon Guzenda) — Sun, 08 Nov 2009 06:01:00 +0000

Many of Objectivity's early customers were implementing workgroup applications. This led to the provision of a crude form of persistent transaction. "Long" transactions could obtain a lock on an object, container or database and have that lock held past the transaction boundary. The lock was tied to a username rather than a process or thread identifier. A subsequent process, perhaps days later, could carry on using the data and eventually release the lock. It's not a frequently used feature, but its still there if you look hard.

One frequent request was for a mechanism to support GUI "undo" operations. The easiest way to do that is to not commit the transaction at the end of a screen interaction. If the next user operation is "Undo" you simply rollback the transaction. If it's anything else you commit the transaction before starting a new one. There is a minor exposure, but users of that kind of application are used to having to explicitly save their data, so they're generally impressed when they find that all but their latest operation has been safely committed to disk, unlike most file-based applications.

An optional mode, that we may add later, would make it possible to handle nested transactions. An inner block can be committed but it may eventually be committed or rolled back by the owning block. The mechanism is generically labelled "Nested Transaction" and it can be very useful in some kinds of application. You can achieve something similar for now by careful use of sessions, thread pools and their associated cache.

The CAD/CAM people were always particularly demanding. Imagine a design application that has an engine block displayed on the screen. The user brings up a new window to create finite element mesh and another one to start generating the numerical control program that will cut the metal. However, the analysis finds a weakness, so the numerical control program generator has to be rolled back, changes made, then the analysis and generation can start again. That's not a nice neat ACID transaction, but it's similar to what might happen if you're working with a spreadsheet and generating multiple charts. You might save some and discard others, having saved the original data one or more times.

So, we made it possible to commit an individual container in the middle of a transaction, but not release the locks.That feature had to be used with great caution, because committing the finite element mess (in the above example) without also committing the design, would cause problems later. This kernel functionality might be best used for something like security audit trails, which can't be rolled back no matter what the user does. In any case, you won't find this feature mentioned anywhere in our documentation any more. It's something our engineers utilize for customers on an as needed basis.

Design applications may also need to share the contents of the cache between threads or processes, perhaps belonging to different users. That mechanism, known as a cooperating transaction, involves assigning a label to a nested transaction and allowing other users to read the contents when the owning transaction is quiescent. It may also be possible to pass an update token around amongst the viewers. This requires the use of in-process shared memory, but we might also implement it with a transient shared cache in the Advanced Multithreaded Server.

You can turn locking off altogether, but that also turns off recovery, so that's something I'd only recommend during development. A more useful mode involves getting a catalog lock, e.g. to declare a whole database as read-only. That can dramatically reduce the lock traffic for a reference database that is being used by a large number of clients.

Lock Server Alternatives

noreply@blogger.com (Leon Guzenda) — Sun, 08 Nov 2009 05:59:00 +0000

Lock Servers are generally stand-alone processes, but you can also link a link server with an application server process. That improves performance for the threads in that process. It is a fully functional lock server, so it will service requests from other processes and client machines.

The Objectivity/DB engineering team has considered several lock server implementations over the past few years. We already support distributed transaction management, e.g. using JCA to commit data into Objectivity/DB and some other database. There has been quite a lot of academic work on user customizable transaction modes that can interoperate safely. Clearly, you can't have two applications having different opinions of what "Exclusive Write Access" means. However, it would allow users to define their own semantics for nested, co-operating and other novel transaction modes.

I'll come back to the lock servers and their role in data replication in a future article. If you have any thoughts or requests regarding locking and transaction management, please let us know by commenting on this topic.

Objectivity Offers An Alternative To Open Source

noreply@blogger.com (Leon Guzenda) — Fri, 30 Oct 2009 17:54:00 +0000

It was interesting to read the news this week that Red Hat has made a financial investment in EnterpriseDB. Red Hat is, of course, the leading example of a company that has made the open source model a financial success. EnterpriseDB is based on the very capable PostgreSQL open source DBMS. It makes sense for Red Hat as they are still locked in a battle with Oracle, which delivers its own version of Linux, for enterprise level sites. Oracle recently acquired MySQL, the most popular open source RDBMS, when it purchased Sun Microsystems.

Open source hasn't been much of a success in the ODBMS market, perhaps because the installed user base is much smaller than the RDBMS base. When db4objects entered the market with their open source ODBMS a few years ago, there was an upsurge in interest in the technology. Unfortunately, despite a large volume of downloads, they never managed to make it to the critical mass needed to sustain their efforts. Versant acquired them for $2.1 million in 2008 (Sun paid about $1 billion for MySQL), so at least the technology has survived, particularly their LINQ implementation, which was pivotal in encouraging the other ODMG members to support LINQ. The Objectivity/DB Release 10 .Net for C# product will support LINQ, underpinned by an enhanced query engine.

Why Objectivity/DB Isn't Open Source

The Objectivity/DB kernel isn't going to be released in open source form any time soon. Quality Assurance for a large body of interoperable, multi-platform code is a very resource-intensive process, requiring a very complex test environment. When we originally scoped out Objectivity/DB, we toyed with the idea of releasing a tool kit of interoperable components, so that developers could select just the pieces needed for a particular application, or even build their own APIs and features. Unfortunately, we couldn't figure out a viable way to test and support a complex product of that type with the tools available at the time (circa 1989) - think "makefile." A few components would appear in any configuration, but we quickly identified hundreds of viable combinations that would have been a nightmare to system test.

Replaceable Components

We've taken an alternative but time-honored approach over the years, adopting the replaceable component model. It leveraged the old "hooks" idea. The first such component appeared in Release 6, when we added support for the industry standard High Performance Storage Systems (HPSS) architecture. We provided two replaceable components - The Objectivity Open File System (OOFS) and the Generalized Security Architecture (GSA) capability. We've added others since then, including the range splitters in the Parallel Query Engine and filters in the query agents.

OOFS allows users to replace the standard filesystem layer with modules of their own. It can be used to interface to novel storage systems or to add specialized caching and streaming capabilities to the Advanced Multithreaded [data] Servers. It also allows users to "piggyback" extra data on the back of messages between a client and the AMS. That can be used to send policy directives, such as "open for random access read usage" to HPSS, for instance. GSA enables users to incorporate their own security mechanisms into AMS, making it possible for local sites to administer and control access to their own data while still collaborating in a federation.

Watch This Space

Objectivity/DB Release 10 will incorporate additional replaceable components that use a standardized "plug-in" architecture. The advantage of this approach, over the pure open source model, is that if users can run the validation tests that we supply with each replaceable module we'll be able to support their deployed applications. We think that it provides most of the flexibility that users have asked for but also enables us to deliver a more stable and predictable product.

We've also started delivering some open source samples in the Developer Network area of our main site. Keep a close an eye on that area as we'll be delivering some very interesting and fully functional modules there in the near future.

Also, we would like to know what kinds of open source applications you would like to see in our Developer Network. Give me your ideas in the comments below.

The Evolution of the ODBMS Market

noreply@blogger.com (Leon Guzenda) — Fri, 23 Oct 2009 16:56:00 +0000

Genesis

When Objectivity, Inc. started in 1988, object-oriented software was still in the early stages, although Smalltalk had a strong following, and C++ was rapidly gaining ground in the telecom and engineering communities. After extensive market research, Objectivity's founders discovered a need for a database geared toward the mechanical and electronic design fields. I'm still amused that Objectivity/DB Release 1 had no discernible query feature, not even predicates in the scan and traversal iterators! That's because the first round of customers didn't need them.

We quickly figured out that priority features for Release 1 included being able to support standard C++ data structures and complex workgroup applications (which meant deciding whether to go distributed or stay with the drawbacks of central servers). I also wanted to make sure that we handled complex relationships as efficiently as possible; with our system, users could find entry points by name, then navigate to related objects or scan batches of data. They applied conditions after we'd returned the objects, rather than before. There was also support for primary keyed objects, using a hashing algorithm to place an object in a specific logical page. That's gone now, but it may come back in a more powerful form.

Launching Release 1

We were worried that the other ODBMS companies rushing to market at that time might "out- tech" us, but that proved not to be the case. Objectivity/DB Release 1 was robust, fully distributed and had more basic features, such as recovery, versioning, naming and relationship management, than its peers.

Four customers built new products based on Release 1 and three of them made it market with products based on Release 1.2, which introduced the Multi-reader, One-Writer concurrency model. The original customers were:

Automation Technology Products (ATP, now Technology Answers,) with the Cimplex solid modeling, mechanical analysis and automatic numerical control program generation package.
Valid Logic Systems (now part of Cadence), with a configuration management package for their ECAD packages.
Matra Data Systems, with a satellite design management application.

The fourth, Sony Corporation, built a prototype product that combined a VCR and a TV games machine. They used Objectivity/DB to hold the game rules and a virtual world. Unfortunately, they'd already started work on the PlayStation and they abandoned the hybrid product as being too expensive.

Interestingly, we spent six months with a deliverable product before we actually made a public announcement about its availability. We let ATP and Valid Logic Systems announce their products at major shows before our first press release. The Quality Assurance paid off (and still does) and the fact that there were already other products out there took a little bit of the risk out of the equation for other evaluators.

Embedded Systems

We spent several years working primarily in the engineering world. Then Digital Equipment Corporation (DEC) started selling Objectivity/DB as DEC ObjectDB, and we broadened our base. Objectivity/DB was embedded in process control equipment, eventually leading to a major deal with Fisher Rosemount. That relationship lives on under the wing of Emerson, in their Delta V range of products.

We also had our first successes in the burgeoning telecom equipment market, first with British Telecom and then with Digital Switch Corporation (now Alcatel). The telecom market developed faster than any other, but we also had a quiet start in the defense market (at Boeing) and in complex financial applications, with Citibank.

Very Large Databases

By the end of the century we were making most of our revenue from the embedded space, primarily telecom, but the Very Large Database market was starting to become significant for us. CERN selected Objectivity/DB for several major projects, including the prototype systems for the Large Hadron Collider. That lead to the highly successful BaBar project at the Stanford Linear Accelerator Center, where we were able to publicly announce that they had built a one-Petabyte database with Objectivity/DB.

Unlike most other large scientific projects, both then and now, they were using the ODBMS as the main repository and directly processing and querying the data. Most projects, including the current LHC project, use a database to index structured files. Their efforts brought Objectivity/DB to the attention of the intelligence and defense communities .

The combination of speed, proven scalability, reliability and cost savings in hardware and in software development effort made it very attractive for advanced projects and systems, which is where more than half of our revenue comes from today.

An Evolution

The lessons we learned from the workgroup, equipment and scientific applications have allowed us to improve Objectivity/DB and its appeal to a broader base of users. We still don't spend much time in conventional IT departments, but that's not our aim. More than 80 percent of the world's data isn't in any kind of structured database. Just about all of that data could be in Objectivity/DB.

By the way, we did add queries at Release 2, mainly in support of Objectivity/SQL++, but we didn't make any major enhancements to that feature until Release 9, with the introduction of the Parallel Query Engine.

Do you have feedback?

You're about to see another major query engine upgrade as a part of Release 10, which some of you have been Beta testing with us. We've also been thinking about Release 11 for a while, so now is a good time to tell us what you'd like to see in it.

What's Old is New Again - Database Sharding

noreply@blogger.com (Leon Guzenda) — Thu, 15 Oct 2009 23:50:00 +0000

There has been a lot of interest lately, from both the relational database and the "NoSQL" communities, in a technique known as sharding, which involves splitting data across multiple devices or servers to improve performance and scalability. Systems such as BigTable, Amazon SimpleDB, Hadoop, MapReduce and Cassandra employ sharding.

Why is it used?

As the number of rows in a relational table grows, performance suffers. This is primarily due to additional index node accesses, but it can also be related to file sizes plus server and disk I/O loads. Likewise, in a distributed file system, the index to particular file can grow to unmanageable sizes.

One solution for Relational Database Management System (RDBMS) users is to split a table into horizontal logical partitions, e.g. use a Country-Name key. If the tables are all stored in a single central server, the tables may need unique names. That complicates queries, as the specific table name has to appear in the SELECT statement. Some RDBMSs support transparent horizontal partitioning, generally allowing segmentation across disk drives. However, partitioning is relatively crude in most RDBMSs. You usually can't group rows from one table with more than one other table in an ad hoc fashion. A Product-Order record might be stored with the Product table, but you couldn't specify the Customer table as the clustering host instead. Sharding can be used to overcome this limitation. Hibernate, the Object-Relational mapping product, has an extension that can make this easier (but still tedious) to manage.

With file-based systems, the client, or a layer of middleware, needs to deal with the sharding, directing file requests to the appropriate node, or to a dedicated service. There's a nice description of the advantages and techniques involved in this article by Curtis Jackon.

The Old New Way

Fortunately, Objectivity/DB has always used sharding as a fundamental component of its distributed, federated, "Single Logical View" architecture. Users can cluster objects into pages in any way that suits the application and data, thereby overcoming the partitioning limitations of RDBMSs. The pages can be clustered into Containers, which can themselves be clustered into Databases. The placement (sharding) strategy can be determined dynamically or be coded into policies. Objectivity/DB's Release 10 will provide enhanced placement management capabilities. Even better, the "Single Logical View" that the federation provides allows transparent navigation of links between objects in different containers or databases, something that is complicated in RDBMS and file-based sharding schemes.

The inherent sharding can be exploited by Objectivity/DB's Parallel Query Engine (PQE). Imagine a federation that has a database for each country in the world. If I want to query all of the databases for countries in Europe, I can build a range splitter that translates "Europe" to the names (or OIDs) of the 49 corresponding databases. PQE will then direct up to 49 parallel queries to the appropriate query agents. If I decide to query the whole federation, PQE can query all of the databases simultaneously and marshal the results.

Flashback/FlashForward

I'm always amused at the way that the computing community continues to reinvent things. Dividing databases up into logical partitions (equivalent to sharding) was a core part of the CODASYL DBTG network database standard way back in 1971. I implemented that standard in the ICL 2900 IDMS product in 1975. The same principle (divide and conquer) influenced the architecture of Objectivity/DB in 1988. Twenty years later, the technique is "hot" again, rechristened "sharding". That reminds me. I should talk about time sharing... I mean "cloud computing"... in a later article.

Welcome to Blog by Objectivity

noreply@blogger.com (Hillary) — Tue, 13 Oct 2009 23:23:00 +0000

I am pleased to welcome you to Objectivity, Inc.'s new blog, Blog by Objectivity. Our goal is to address the many trends and developments taking shape in the data management field today.

For more than two decades, our company has had front-row access to every twist, turn and triumph affecting the database industry. During that time, data management has emerged as one of the most pervasive issues facing organizations. More data is being generated from more sources than ever before, and companies are under intense pressure to manage data quickly and efficiently, without resorting to huge capital expenditures.

The challenges may be overwhelming, but we plan to provide information and resources to help people working at all levels of data management to find the solutions they need.

One of the best people to help guide you through the process is our founder, Leon Guzenda. Leon was on the team that launched Objectivity, Inc. in 1988, and he continues to play a critical role as both a technical adviser and market specialist. Leon will be the primary author of this blog, and he will contribute commentary not only about our niche market, but also about industry news that affects our current and future customers. You can check out his full list of credentials here.

We are excited to start a discussion with our customers, peers, evaluators, friends, and anyone else who may be interested in data management. Please feel free to leave comments or suggestions. We look forward to hearing from you.

Hillary Schuler-Jones
Editor, Blog by Objectivity