Data Fusion - Case Study 3
Data Fusion - integrating complex data from multiple sources
CASE STUDY 3 – High Performance Computing
A Massive Data Acquisition, Number Crunching and Information Dissemination System
The Stanford Linear Accelerator Center [SLAC] BaBar Database System provides data persistency for the BaBar project. It is probably the largest unclassified database in the World. The vital statistics are impressive:
- There are over 5 million lines of custom written C++ code.
- There are over 2000 CPUs and 100 lock/data servers.
- There is over 750 Terabytes of data within the Objectivity/DB databases, stored in more than 45,000 database files.
- Data is shared by 76 organizations in North America and Europe.
The data is complex, with many variable length arrays and relationships. Each accelerator event requires many teraflops of compute power to render the raw data into information that is useful to the physicists.
This system is directly analogous to the pattern seen in many commercial and intelligence systems. There are three stages in the pipeline: a raw data input stage; a number crunching and data correlation stage; and a data interpretation and dissemination stage.