D0 Database Meeting Minutes May-22-2006

General Database and DB Server

DBA minutes this week:

• May 22, 2006 DBA notes

Generic db server reminder

I forgot to include this in the last minutes: According to Kin Yip, if there are db server problems: The user should write D0 SAM Admin to ask to restart the DB server or investigate the problems. Also cc-ing D0 Database Support does not hurt.

Problem with trigger db server

On May 20 and 21 (Sunday), Arnold Pompos and Karel Soustruz reported fatal errors to d0d0-support@fnal.gov connecting the trigger db server.

Steve White investigated Sunday (21-May) and found:

"The disks on d0ora2 filled up. The fix required the SAM expert to restart the naming service without the use of its IOR log file. This caused the name service to lose track of the trigger dbserver (and everything else that was registered to it). I've restarted the dbserver, you should be ok now."

ALL dbservers and everything else in the nameservice were affected by this. The SAM shifter restarted the SAM dbservers and Calibration dbservers.

The Trigger dbserver was forgotten.

It would be helpful if the trigger db client db server error would tell the user to contact D0 SAM Admin and D0 Database Support whenever there is a problem connecting to any db server.

Problem with user db servers

On Friday May 19, Angela Bellevance reported similar errors coming from the db servers as we saw last week ("%ERLOG-e UNKNOWN: d0omCORBA: Failed to ping server D0DbServer.user_prd CERR 19-May-2006 10:21:55 CDT calibration_management:calib Run number: 201670, Event Number:713822"

Robert restarted the servers. Steve Kovich is investigating machine problems.

Since this has happened twice in the last 2 weeks (dbserv5 hung), Steve increased the number of 'interrupts' from 20 to 30. If this increase does not resolve this issue, then the fail over should be improved to use something other than a ping to assess server status.

Upgrade of offline databases to Oracle 10gR2:

The d0ofdev1 instance was upgraded to Oracle 10gR2 tuesday 9-May so testing can now proceed.
REMINDER: May 31 - deadline for sign off or submit error reports
All applications must complete 10g testing this week !! (since this coming Monday is a holiday).
• Calibration:
  Taka reported his test results of Oracle 10g on SMT db servers:
  "I'm testing the calibration database right now. It is very slow. It is taking 2 min to query one subset of SMT data. I need to query 1096 SMT subset to get an entire SMT set. It will take more than 3 hours to query one SMT set. It is about 5 min on the current production database. Please investigate why. "
  Anil found that the indexes did not get defined. Taka will send DBA's the ddl to define the indexes so that Taka can test again.

Luminosity

Planning for Luminosity constant change

• Nelly imported all the data from d0oflump to d0olumd, but found the index definitions were taking a really long time. We decided to disable reads to d0oflumd until the import is complete since these reads (users unknown) were found to be causing the slowness.
• Elizabeth disabled the apache link of the development operations reports to d0oflumd.
• When the import is complete, Nelly will send Elizabeth and Dan email to put the development Operations Reports back online and reload the DQ data so that DQ testing can resume.
• Elizabeth is developing the test script to create, add, delete, rename columns for the DBA's suggest Elizabeth add data to the tables to make the test more realistic. Elizabeth will create the tables on d0ofdev1 under username gallas.