The Fastest Platform on the Planet  for Connections and Intelligence

The Fastest Platform on the Planet for Connections and Intelligence

Introduction Successful Big and Fast Data projects require a platform that can:             Scale massively             Perform in near real-time The Objectivity ThingSpan platform is unique in its ability to meet those needs. No other platform comes close. As an example, this blog highlights one use case that far exceeds anything else available.   This example also illustrates ThingSpan’s ability to grow linearly as computing resources are added. Further, it shows the value of ThingSpan’s user language, DO, to aid the user in analyzing and visualizing the desired results. Finally, this example is available in the Amazon Cloud, so that you can confirm ThingSpan’s performance and scale for yourself.   Does today’s massive quantities of data from multiple sources fit neatly into rows and tables for enterprises and governments to analyze and get the answers needed to make critical operational decisions in near real-time?  Have open source technologies proven that they can handle the performance and scale demanded of mission critical operational systems? Can critical operations afford to wait for hours and days for after the fact analytics?   Objectivity’s ThingSpan is the only data management platform that can ingest billions of complex, interrelated records while simultaneously building the graph in order to gain new insights in near real-time for operational systems.  The use case further described below is a very simple one for ThingSpan yet the customer’s existing relational based system was not even able to provide an answer back.  ThingSpan not only ingested the data 5 times faster while also connecting the data, but was able to perform complex queries that their existing relational system could not...
ThingSpan Performance Blog Series – Part III

ThingSpan Performance Blog Series – Part III

Graph Analytics with Billions of Daily Financial Transaction Events   Introduction In this third blog of the series we updated the performance results to include the numbers from running on a 64 node Amazon EC2 cluster. Also, we continue to look for different ways to optimize the performance of ingest and query. In this update, we further optimized the way ThingSpan stores relationships for large degree fan out connectivity. As a reminder, the requirement for this use case was to ingest one billion financial transaction events in under 24 hours (about 12,000 transactions a second) while being able to perform complex query and graph operations simultaneously. ThingSpan has proven to surpass this requirement to ingest while performing these complex queries at the same time at speed and scale. It is important to note that the graph in this case is incrementally added to as new events are ingested. In this way the graph is always up to date with the latest data (nodes) and connections (edges) and is always available for query in a transactional consistent state, what we call a living graph. We used Amazon EC2 nodes for the tests and found that “m4x2large” gave the best resources for scalability, performance, and throughput. We initially surpassed the performance requirements using a configuration consisting of 16 “m4x2large” nodes. For this test we used the native Posix file system. Full details of the proof of concept can be downloaded from: ThingSpan in Financial Services White Paper   Updated Results Below are the updated results when running on a 64 node Amazon EC2 cluster, with the performance optimization described above. We...
ThingSpan Performance Blog Series – Part II

ThingSpan Performance Blog Series – Part II

Graph Analytics with Billions of Daily Financial Transaction Events   Introduction In this second blog of the series we updated the performance results to include the numbers from running on a 32 node Amazon EC2 cluster. As a reminder, the requirement for this use case was to ingest one billion financial transaction events in under 24 hours (about 12,000 transactions a second) while being able to perform complex query and graph operations simultaneously. ThingSpan has proven to handle this requirement to ingest while performing these complex queries at the same time at speed and scale. It is important to note that the graph in this case is incrementally added to as new events are ingested. In this way the graph is always up to date with the latest data (nodes) and connections (edges) and is always available for query in a transactional consistent state, what we call a living graph. We used Amazon EC2 nodes for the tests and found that “m4x2large” gave the best resources for scalability, performance, and throughput. We surpassed the performance requirements using a configuration consisting of 16 “m4x2large” nodes. For this test we used the native Posix file system. Full details of the proof of concept can be downloaded from: ThingSpan in Financial Services White Paper   Results Below are the updated results when running on a 32 node Amazon EC2 cluster. We can use these results to show near linear scale up as the volume of data increases and scale out to achieve better throughput performance. The blue bar represents the time to ingest 250 million events.  Therefore, by doubling the number of...
ThingSpan Performance Blog Series – Part I

ThingSpan Performance Blog Series – Part I

Graph Analytics with Billions of Daily Financial Transaction Events   Introduction In this first blog of the series we will explain how ThingSpan can deliver solutions to meet the demanding needs of today’s complex analytics systems by combining big and fast data sources. In particular, we will focus on ThingSpan’s capabilities to scale up as data volumes increase, and scale out to meet response time requirements. The requirement for this use case was to ingest one billion financial transaction events in under 24 hours (about 12,000 transactions a second) while being able to perform complex query and graph operations simultaneously. ThingSpan has proven to handle this requirement to ingest while performing these complex queries at the same time at speed and scale. It is important to note that the graph in this case is incrementally added to as new events are ingested. In this way the graph is always up to date with the latest data (nodes) and connections (edges) and is always available for query in a transactional consistent state, what we call a living graph. We used Amazon EC2 nodes for the tests and found that “m4x2large” gave the best resources for scalability, performance, and throughput. We surpassed the performance requirements using a configuration consisting of 16 “m4x2large” nodes. For this test we used the native Posix file system. Full details of the proof of concept can be downloaded from: ThingSpan in Financial Services White Paper   Test Description Data In our example use case, approximately one billion financial transaction events occur each day. Each event produces a subgraph that creates or merges 8-9 vertices and creates...
In the beginning

In the beginning

In the beginning there was data. Then Codd (and Date) created relational database systems, and then there was structured query language (SQL). SQL was good for queries by values of data, and queries where you knew what you were looking for. You could answer the known questions. Data was neatly organized into rows (records) and columns (fields) of tables. You could even query across tables using “joins” if you knew what to join.