Data Fusion - Case Study 1
Data Fusion - integrating complex data from multiple sources
CASE STUDY 1 - Indexing Multiple Data Sources
A Consolidated Credit Union Member Database: In January of 1998 Credit Union of North America [CUNA] set out to help its associated credit unions offer better services to their members by creating a consolidated member database. Previously data was spread out over disparate, policy number keyed databases rather than one with a member focus. CUNA had spent over $1 Million on a previous attempt to use an RDBMS to solve the problem, but had experienced huge difficulties in mapping the required object model to the legacy technology.
Complexity - The data consisted of over 50 million records from over 5000 sources. There were over 4000 file formats representing over 30 million individuals. The database has since grown to over 100 million records and is over one terabyte in size. CUNA used parallel processing to load 18,000 objects and hash table entries per second into multiple Objectivity Databases in a single Federated Database.
Performance - The initial task was to load the records and create sorted proxies that reference the original data source. A sophisticated rule-based match and selection process refined the proxies for business object creation, determining, for example, which of John Doe’s addresses is the most recently verified. The final step was to create multi-level hierarchies that allow the user to view increasing levels of detail. Member lookup times average around one millisecond.