Data Fusion - Case Study 6
Data Fusion - integrating complex data from multiple sources
CASE STUDY 6 –A Parallel Search Engine
Exploiting the Objectivity/DB Storage Architecture
The time that it takes to search for an unindexed object in a very large database is governed by the speed of the I/O hardware. Sequentially scanning a 100 Terabyte federation for a single unindexed object could take FOUR YEARS! However, a fairly simple modification to Objectivity/DB opens up the possibility of parallel scans of only those containers known to hold objects of the required type.
A Parallel Query Engine would look at system information to rapidly determine the identities of containers that might contain instances of the required object. It would then use a user configurable number of threads, probably running close to the appropriate page servers, to iterate over the objects in each of the target containers. In an ideal case the number of threads would equal the number of containers. For simplicity, the diagram illustrates a case where there is only one target container “behind” each Page Server. In practice there would probably be many and there would be more query threads.
In the example cited above, the search time for the sole object of a given class, residing anywhere in the federation, would be reduced to around FOUR SECONDS [assuming 50,000,000 containers, database file sizes around 2 Gb and adequate hoarware].
