% ----------------------------------------------------
% Sent to Gavin week of Aug 15, 2005:
% from Taka and Elizabeth
% ----------------------------------------------------
\chapter{Databases}
\label{chap:dbs}

\section{Offline Calibration Database}

The calibration constants for the SMT, CFT, CPS, FPS and Calorimeter are served
to the reconstruction
processes on the Fermilab reconstruction farm and to the jobs that users run for
their own purposes via the DAN client-server system based on the CORBA technology
and developed by the Computing Division  and D\O .
This system can also deliver calibration constants for the
muon subsystems to {\it d0reco}.
Up to now the muon group has not used this facility for {\it d0reco} processing, but
is planning to use it in the near future, with updated database tables.

The database has been in use since June, 2003 and is fully reliable. There is one
dedicated linux node for running the calibration DB servers for the farm and
another for the non-farm users. Since these nodes are not supported 24/7,
there is a failover node and a procedure in case of troubles with the server nodes.
However these nodes are three year old
machines and will need to be replaced in the near future.

The D\O\ farm runs 600 processes and requests a new set of calibration constants
on average every 100 seconds. It takes about 90 seconds of wall-clock time for
a {\it d0reco} process to receive one whole set from the servers, including the time for
marshalling data on the servers and unmarshalling them on the client.
The cpu usage of the farm server node is about 5-10 \% and the network usage out
of the server node is about 30 \%. This implies that up to
a three-fold increase in the number of the farm nodes can be handled
without upgrading the network interfaces.

The p17 remote reprocessing started on March 25, 2005 and the calibration constants
have been delivered to the remote processes through the remote proxy servers since then.
The perticipating institutions on the remote processing are GridKa, Imperial College,
IN2P3, SAR and Westgrid.
The system is reliable and we have not had problems with serving calibration constants
to the remote institutions.

One possible future problem with the system is that the interface to the Oracle database
is based on a freeware python package which is no longer supported by the developer.
It is preferrable to be prepared to switch to a different interface that is supported.

\section{Luminosity Database}

Many existing components of the Luminosity software are based
   on ASCII files, a system which will become unmanageable
   in the life span of the experiment.  In order to address this issue for the long term,
   the D0 Luminosity Group and the D0 Database Group have been
   developing the Luminosity Database Application.
Transferring the existing information to an Oracle database
   will allow us to profit from the better tools available
   with a modern database system.

This Application is a large scale and complex project which,
   at it's core, relies upon a set of 37 database tables and 2 views
   stored in a new offline production database instance
   on a new machine that is now being set up.
The Database itself contains information from many subsystems:
   the Luminosity Monitor Detector,
   the Trigger Framework,
   the Level 3 system,
   the Fermilab Tevatron Accelerator and information about
   Offline Data Processing (from SAM).
It it will also contain "Group" assigned data quality and
   "Consistency" determined data quality.

The Application includes components that will satisfy the following
essential operations for the D0 experiment:

\begin{itemize}
\item Calculate and deliver a Luminosity Normalization (lmAccess) for
   any given data sample
   (replacing the existing file based system)
\item Improve error tracking and characterization by making
   fundamental luminosity information available to offline programs
\item Produce Operations Reports and Graphs
   (replacing the existing file based system)
\item Produce LBN Quality Reports
   (replacing the existing Run based data quality)
\item Verify and certify data integrity for normalization purposes

\item Additional customized reports also become possible using
    this system.
\end{itemize}

The database schema has been stable for a year now.
Nearly all of the data taken by the experiment through 2004
   has been loaded into database tables.  That data will be transferred to the production machine when
   the machine is available.  We are in the process of loading data for 2005.
The application that loads the data from the online system as the
   data is recorded is currently in development.

With much of the data loaded,
   large scale testing of all the client applications
   (the components of the system itemized above)
   is nearly completed.
More detailed testing is underway with realistic operations
   use cases and actual analyses to bring each component
   to 'production' quality.

To access the luminosity database from the analysis programs,
   we have developed client-server system similar to the calibration system.
The database server was generated using the same code generation system
   as the calibration system,
   and the client code called lm\_access has been converted to
   use the CORBA protocol instead of reading the ASCII files.
At this time, the new lm\_access performs as fast as or faster than the old system.

We plan to move this Application into production before the end of FY05.

\section{Trigger Database}

The Trigger database stores trigger configuration parameters and settings
   entered by trigger experts
   including information from all trigger subsystems
   at each level of the trigger.
These settings are downloaded at Run time
   for physics, test and calibration data taking
   and used by the trigger simulator.
The system includes interfaces
   to report this information
   to the hundreds of physicists
   in the collaboration who are involved in optimizing data collection
   and/or performing offline physics analysis.

The Trigger Database Application
   has been in use since the start of Run II.
The TDB Entry system and some command line interfaces rely on
   a database server that is similar to the SAM servers
   (server functions include some business logic).

In the last year,
   with personnel from the CD Database Software and the
   CD Run 2 Computing and Analysis Groups,
   major progress has been made to bring the web client and
   database server components to completion
   (requiring less day to day maintenance)
   and to serve the experiment more efficiently.
The progress has been facilitated and organized
   according the Trigger Database Project Plan
   which describes all components of the
   application and includes a detailed, prioritized
   description of the work that needs to be completed.
Work according to the Plan is an ongoing effort which should be
   completed by the end of the calendar year.

New trigger systems will come online in Run 2b
   and many of the new trigger features
   will be programmable through the existing system
   without changes to the database, client or server
   (eased by the completion of the Project Plan).
Changes to the system may be required for some
   new trigger features,
   but how those features will be configured
   has not been completely specified.

