We are at the cusp of the golden age of Fast Data. As more and more organizations deploy massive sensor networks, there is a growing imperative to better leverage increasing volumes of data from sensor networks to provide timely insight and improve business operations. At the same time, a recent McKinsey & Co report on the Internet of Things (“Internet of Things: Mapping the Value Beyond the Hype”) points out that in some industries, less than 1% of data from sensor networks is being utilized in discovering operational insights. This chasm between potential and real value can be bridged through improved streaming analytics tools, data fusion tools, and Hadoop-based enterprise data management systems to allow organizations to pursue new data strategies for creating greater value from IoT Fast Data streams.
In the course of my professional journey across the evolving landscape of Big Data and Fast Data, I have encountered many organizations seeking to better leverage the potential of IoT data streams but running into two types of problems.
First is the lack of solutions or solution patterns for developing operational context from IoT data streams and static data in other operational data systems. This is because sensor data streams are not well suited for analysis in raw form. Typically, raw sensor data needs to be transformed into appropriate datasets to support specific IoT analytic workloads based on both streaming data and other static data, such as historical and transactional information as part of the fusion process.
Fusion operators, such as filters, aggregation, correlation, and clustering, transform sensor data into target datasets and generate associated metadata to support the analytic workload. The operational context consisting of transformed datasets and associated metadata is needed to support analytic engines that are at the core of new IoT applications. As a result, many organizations resort to “composition approach” to building IoT applications by taking various components, such as NoSQL or NewSQL databases, messaging tools, such as Kafka, streaming tools, etc. to build one-off solutions. This requires a high level of domain expertise.
The second type of problems involves a lack of expertise on how to build a Big Data analytic stack from a highly dynamic list of components. As a result, many organizations struggle to implement the right Big Data stack to support analytics for IoT data. These two issues often lead to long time-to-production of IoT applications involving high risk and expenses.
One of the reasons I joined Objectivity was because of the experience the company and its people had in helping our customers over the years more effectively realize better, faster value from their Fast Data streams using data fusion methodology based on object data-modeling concepts. Translating this success into product and partner strategy that integrates and complements the emerging Big Data and Fast Data ecosystems was an exciting opportunity for me.
Objectivity’s strategy for addressing the IoT data challenge facing many organizations involves both product and partner components. The first example of this strategy in action is our announcement of our ThingSpan product and our support of Intel’s Trusted Analytics Platform as a partner. ThingSpan is an advanced information fusion platform that packages our experience and technology for easier integration and consumption by helping government and Fortune 1000 companies deploy advanced fusion solutions involving Fast Data.
ThingSpan utilizes object-data modeling technology and data fusion methods to support fusion patterns involving fast streaming data and domain-specific static data to enable the creation of context in real time in the form of transformed data and domain metadata to support IoT application workloads. By supporting advanced fusion patterns within the Hadoop 2.0 and Spark environment, ThingSpan enables organizations to leverage the benefits of scale-out computing infrastructure and all of the dynamic components of the Hadoop ecosystem to accelerate the time-to-production of IoT applications.
Our support of Intel’s TAP is part of a broader partner strategy to help our customers deploy advanced Big Data analytic systems faster. TAP is an open-source software, optimized for performance and security that is intended to accelerate the creation of applications driven by Big Data analytics. TAP provides reference architecture for Objectivity’s customers to quickly build out their Big Data analytic stack as part of their effort to better leverage their Fast Data streams.
ThingSpan integrates with TAP-based systems as part of the IoT streaming data ingest pipeline to create the datasets and associated metadata needed to support modern IoT applications. TAP was designed from Intel’s experiences in working with some of the most complex Big Data problems involving Fast Data. As a result, it is well-suited to be the Big Data backbone of organizations implementing IoT application system environments to better leverage their sensor data streams.
As the volume and complexity of Fast Data from massive sensor networks increase from industries as diverse as Oil & Gas and Energy to Healthcare and Logistics, we have an opportunity to dramatically increase the productivity of various industries. Using IoT sensor data more effectively could yield a multi-trillion dollar benefit to the world economy, according to McKinsey & Co. and other similar reports on IoT. The emerging streaming, fusion, and Big Data technologies are helping organizations bridge the chasm between the potential benefits of IoT and real, measurable solutions.
VP of Marketing and Partner Development